[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:07:32 dist-test-slave-8pgz krb5kdc[15192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:07:32 dist-test-slave-8pgz krb5kdc[15192](info): set up 2 sockets
Oct 26 15:07:32 dist-test-slave-8pgz krb5kdc[15192](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20251026 15:07:34.303570 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.985s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:07:34 dist-test-slave-8pgz krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:07:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:07:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:34.438823 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33997
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:33997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:34.518811 15208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:34.519011 15208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:34.519049 15208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:34.520501 15208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:07:34.520550 15208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:34.520572 15208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:34.520593 15208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:07:34.520613 15208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:07:34.522253 15208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:33997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:34.522531 15208 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:34.522727 15208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:34.525739 15214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:34.525784 15216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:34.525717 15213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:34.526000 15208 server_base.cc:1047] running on GCE node
I20251026 15:07:34.526281 15208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:34.526654 15208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:34.527819 15208 hybrid_clock.cc:648] HybridClock initialized: now 1761491254527798 us; error 42 us; skew 500 ppm
Oct 26 15:07:34 dist-test-slave-8pgz krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491254, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:34.530409 15208 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:07:34.530956 15208 webserver.cc:492] Webserver started at http://127.14.212.126:36389/ using document root <none> and password file <none>
I20251026 15:07:34.531147 15208 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:34.531186 15208 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:34.531301 15208 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:34.532238 15208 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "7338702966774639a4f41494cfe56aab"
format_stamp: "Formatted at 2025-10-26 15:07:34 on dist-test-slave-8pgz"
server_key: "1e4a1dc44a60f6679b33b7d4ccbe5804"
server_key_iv: "b971ff223806435adb04fc0c4b032812"
server_key_version: "encryptionkey@0"
I20251026 15:07:34.532595 15208 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "7338702966774639a4f41494cfe56aab"
format_stamp: "Formatted at 2025-10-26 15:07:34 on dist-test-slave-8pgz"
server_key: "1e4a1dc44a60f6679b33b7d4ccbe5804"
server_key_iv: "b971ff223806435adb04fc0c4b032812"
server_key_version: "encryptionkey@0"
I20251026 15:07:34.533787 15208 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:07:34.534483 15223 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:34.534636 15208 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:34.534698 15208 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "7338702966774639a4f41494cfe56aab"
format_stamp: "Formatted at 2025-10-26 15:07:34 on dist-test-slave-8pgz"
server_key: "1e4a1dc44a60f6679b33b7d4ccbe5804"
server_key_iv: "b971ff223806435adb04fc0c4b032812"
server_key_version: "encryptionkey@0"
I20251026 15:07:34.534762 15208 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:34.549237 15208 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:34.550969 15208 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:34.551097 15208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:34.555047 15208 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:33997
I20251026 15:07:34.555078 15275 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:33997 every 8 connection(s)
I20251026 15:07:34.555392 15208 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:07:34.555971 15276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:34.558306 15276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab: Bootstrap starting.
I20251026 15:07:34.558889 15276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:34.559119 15276 log.cc:826] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:34.559821 15276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab: No bootstrap required, opened a new log
I20251026 15:07:34.561056 15276 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338702966774639a4f41494cfe56aab" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33997 } }
I20251026 15:07:34.561189 15276 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:34.561213 15276 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7338702966774639a4f41494cfe56aab, State: Initialized, Role: FOLLOWER
I20251026 15:07:34.561347 15276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338702966774639a4f41494cfe56aab" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33997 } }
I20251026 15:07:34.561414 15276 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:07:34.561450 15276 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:07:34.561503 15276 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:34.562072 15276 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338702966774639a4f41494cfe56aab" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33997 } }
I20251026 15:07:34.562208 15276 leader_election.cc:304] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7338702966774639a4f41494cfe56aab; no voters:
I20251026 15:07:34.562359 15276 leader_election.cc:290] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:07:34.562436 15279 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:34.562561 15279 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [term 1 LEADER]: Becoming Leader. State: Replica: 7338702966774639a4f41494cfe56aab, State: Running, Role: LEADER
I20251026 15:07:34.562639 15276 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:07:34.562716 15279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338702966774639a4f41494cfe56aab" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33997 } }
I20251026 15:07:34.563102 15280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7338702966774639a4f41494cfe56aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338702966774639a4f41494cfe56aab" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33997 } } }
I20251026 15:07:34.563129 15281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7338702966774639a4f41494cfe56aab. Latest consensus state: current_term: 1 leader_uuid: "7338702966774639a4f41494cfe56aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338702966774639a4f41494cfe56aab" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33997 } } }
I20251026 15:07:34.563184 15280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [sys.catalog]: This master's current role is: LEADER
I20251026 15:07:34.563211 15281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab [sys.catalog]: This master's current role is: LEADER
I20251026 15:07:34.563270 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 15208
I20251026 15:07:34.563366 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:07:34.564136 15185 external_mini_cluster.cc:1468] Setting key 346037ee604adc4db1199dfee694722e
I20251026 15:07:34.564155 15285 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:07:34.564692 15285 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:07:34.566051 15285 catalog_manager.cc:1357] Generated new cluster ID: 86c0b0172301407cbda41f56eb4ac155
I20251026 15:07:34.566095 15285 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 26 15:07:34 dist-test-slave-8pgz krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:34.577697 15285 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:07:34.578289 15285 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:07:34.578521 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:34.565235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.14.212.126:33997)
1026 15:07:34.565402 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:34.565413 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:34.565440 (+ 27us) server_negotiation.cc:408] Connection header received
1026 15:07:34.566928 (+ 1488us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:34.566934 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:34.566982 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:34.567123 (+ 141us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:34.568646 (+ 1523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:34.569827 (+ 1181us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:34.570575 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:34.570824 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:34.574452 (+ 3628us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:34.574458 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:34.574459 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:34.574479 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:34.576574 (+ 2095us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:34.577048 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:34.577051 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:34.577051 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:34.577110 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:34.577343 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:34.577344 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:34.577344 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:34.577600 (+ 256us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:34.577747 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:34.578129 (+ 382us) server_negotiation.cc:300] Negotiation successful
1026 15:07:34.578253 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20251026 15:07:34.582273 15285 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7338702966774639a4f41494cfe56aab: Generated new TSK 0
I20251026 15:07:34.582435 15285 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:34.626803 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33997
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:34.706059 15304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:34.706247 15304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:34.706283 15304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:34.707718 15304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:34.707765 15304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:34.707806 15304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:07:34.709393 15304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:33997
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:34.709628 15304 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:34.709813 15304 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:34.712327 15309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:34.712332 15312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:34.712332 15310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:34.712713 15304 server_base.cc:1047] running on GCE node
I20251026 15:07:34.712881 15304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:34.713075 15304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:34.714236 15304 hybrid_clock.cc:648] HybridClock initialized: now 1761491254714219 us; error 31 us; skew 500 ppm
Oct 26 15:07:34 dist-test-slave-8pgz krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491254, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:34.717016 15304 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:07:34.717689 15304 webserver.cc:492] Webserver started at http://127.14.212.65:44475/ using document root <none> and password file <none>
I20251026 15:07:34.717908 15304 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:34.717960 15304 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:34.718073 15304 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:34.719323 15304 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "cd1e04624774413fb9dcc65b3177567d"
format_stamp: "Formatted at 2025-10-26 15:07:34 on dist-test-slave-8pgz"
server_key: "d8a32361173539cbeb5d88240f6e7f10"
server_key_iv: "8733375ffc0a9853b5d1ffb18a879c6b"
server_key_version: "encryptionkey@0"
I20251026 15:07:34.719658 15304 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "cd1e04624774413fb9dcc65b3177567d"
format_stamp: "Formatted at 2025-10-26 15:07:34 on dist-test-slave-8pgz"
server_key: "d8a32361173539cbeb5d88240f6e7f10"
server_key_iv: "8733375ffc0a9853b5d1ffb18a879c6b"
server_key_version: "encryptionkey@0"
I20251026 15:07:34.721086 15304 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:07:34.721827 15319 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:34.722002 15304 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:34.722064 15304 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "cd1e04624774413fb9dcc65b3177567d"
format_stamp: "Formatted at 2025-10-26 15:07:34 on dist-test-slave-8pgz"
server_key: "d8a32361173539cbeb5d88240f6e7f10"
server_key_iv: "8733375ffc0a9853b5d1ffb18a879c6b"
server_key_version: "encryptionkey@0"
I20251026 15:07:34.722126 15304 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:34.735306 15304 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:34.736960 15304 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:34.737064 15304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:34.737238 15304 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:07:34.737545 15304 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:34.737577 15304 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:34.737600 15304 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:34.737613 15304 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:34.743399 15304 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:35721
I20251026 15:07:34.743479 15433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:35721 every 8 connection(s)
I20251026 15:07:34.743726 15304 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
Oct 26 15:07:34 dist-test-slave-8pgz krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491254, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:34.751406 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 15304
I20251026 15:07:34.751498 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:07:34.751600 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:34.744351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:55519 (local address 127.14.212.126:33997)
1026 15:07:34.744409 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:34.744411 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:34.744535 (+ 124us) server_negotiation.cc:408] Connection header received
1026 15:07:34.745023 (+ 488us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:34.745024 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:34.745033 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:34.745092 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:34.745887 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:34.746368 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:34.746967 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:34.747090 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:34.749278 (+ 2188us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:34.749282 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:34.749283 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:34.749289 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:34.750540 (+ 1251us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:34.750887 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:34.750888 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:34.750888 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:34.750914 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:34.751056 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:34.751056 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:34.751056 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:34.751139 (+ 83us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:34.751181 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:34.751400 (+ 219us) server_negotiation.cc:300] Negotiation successful
1026 15:07:34.751505 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251026 15:07:34.751690 15185 external_mini_cluster.cc:1468] Setting key f289094b3d1f13e1c177a20e2544553a
I20251026 15:07:34.751670 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:34.744299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33997 (local address 127.14.212.65:55519)
1026 15:07:34.744435 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:34.744465 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:34.744954 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:34.745193 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:34.745197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:34.745505 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:34.745815 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:34.745818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:34.746468 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:34.746469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:34.746893 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:34.746895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:34.747005 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:34.747889 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:34.747904 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:34.749186 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:34.750609 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:34.750614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:34.750616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:34.750825 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:34.750963 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:34.750964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:34.750964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:34.751013 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:34.751181 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:34.751183 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:34.751289 (+ 106us) client_negotiation.cc:769] Sending connection context
1026 15:07:34.751366 (+ 77us) client_negotiation.cc:240] Negotiation successful
1026 15:07:34.751463 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20251026 15:07:34.752207 15434 heartbeater.cc:344] Connected to a master server at 127.14.212.126:33997
I20251026 15:07:34.752319 15434 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:34.752508 15434 heartbeater.cc:507] Master 127.14.212.126:33997 requested a full tablet report, sending...
I20251026 15:07:34.752931 15240 ts_manager.cc:194] Registered new tserver with Master: cd1e04624774413fb9dcc65b3177567d (127.14.212.65:35721)
I20251026 15:07:34.753520 15240 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:55519
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:34.794525 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33997
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:34.872063 15441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:34.872239 15441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:34.872262 15441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:34.873605 15441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:34.873642 15441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:34.873674 15441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:07:34.875221 15441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:33997
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:34.875403 15441 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:34.875540 15441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:34.878093 15447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:34.878190 15441 server_base.cc:1047] running on GCE node
W20251026 15:07:34.878085 15446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:34.878110 15449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:34.878499 15441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:34.878783 15441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:34.879971 15441 hybrid_clock.cc:648] HybridClock initialized: now 1761491254879962 us; error 36 us; skew 500 ppm
Oct 26 15:07:34 dist-test-slave-8pgz krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491254, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:34.882416 15441 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:07:34.882957 15441 webserver.cc:492] Webserver started at http://127.14.212.66:37605/ using document root <none> and password file <none>
I20251026 15:07:34.883138 15441 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:34.883177 15441 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:34.883291 15441 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:34.884215 15441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "c0ff651bbc7749adb47623a6d7f2afed"
format_stamp: "Formatted at 2025-10-26 15:07:34 on dist-test-slave-8pgz"
server_key: "4257da4718c0aaec65629d8adfb12f58"
server_key_iv: "5e0045f911ed2b4582a244044f8c4c67"
server_key_version: "encryptionkey@0"
I20251026 15:07:34.884524 15441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "c0ff651bbc7749adb47623a6d7f2afed"
format_stamp: "Formatted at 2025-10-26 15:07:34 on dist-test-slave-8pgz"
server_key: "4257da4718c0aaec65629d8adfb12f58"
server_key_iv: "5e0045f911ed2b4582a244044f8c4c67"
server_key_version: "encryptionkey@0"
I20251026 15:07:34.885702 15441 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:07:34.886361 15457 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:34.886528 15441 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:34.886592 15441 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "c0ff651bbc7749adb47623a6d7f2afed"
format_stamp: "Formatted at 2025-10-26 15:07:34 on dist-test-slave-8pgz"
server_key: "4257da4718c0aaec65629d8adfb12f58"
server_key_iv: "5e0045f911ed2b4582a244044f8c4c67"
server_key_version: "encryptionkey@0"
I20251026 15:07:34.886653 15441 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:34.896983 15441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:34.898931 15441 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:34.899065 15441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:34.899263 15441 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:07:34.899557 15441 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:34.899590 15441 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:34.899623 15441 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:34.899653 15441 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:34.905295 15441 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:40799
I20251026 15:07:34.905361 15572 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:40799 every 8 connection(s)
I20251026 15:07:34.905629 15441 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:07:34.908385 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 15441
I20251026 15:07:34.908473 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:07:34.908623 15185 external_mini_cluster.cc:1468] Setting key 687df06d32ea80c64f48b7a0f59b0572
Oct 26 15:07:34 dist-test-slave-8pgz krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491254, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:34.915032 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:34.906324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:50785 (local address 127.14.212.126:33997)
1026 15:07:34.906380 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:34.906383 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:34.906550 (+ 167us) server_negotiation.cc:408] Connection header received
1026 15:07:34.907082 (+ 532us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:34.907083 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:34.907098 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:34.907152 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:34.907959 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:34.908587 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:34.909198 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:34.909364 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:34.911807 (+ 2443us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:34.911811 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:34.911811 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:34.911819 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:34.913672 (+ 1853us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:34.914076 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:34.914078 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:34.914078 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:34.914106 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:34.914275 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:34.914276 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:34.914276 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:34.914422 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:34.914492 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:34.914816 (+ 324us) server_negotiation.cc:300] Negotiation successful
1026 15:07:34.914915 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251026 15:07:34.915063 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:34.906269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33997 (local address 127.14.212.66:50785)
1026 15:07:34.906444 (+ 175us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:34.906474 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:34.906977 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:34.907216 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:34.907219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:34.907539 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:34.907857 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:34.907860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:34.908680 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:34.908682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:34.909130 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:34.909132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:34.909249 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:34.910240 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:34.910254 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:34.911726 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:34.913774 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:34.913779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:34.913782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:34.914004 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:34.914167 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:34.914168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:34.914168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:34.914223 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:34.914510 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:34.914515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:34.914686 (+ 171us) client_negotiation.cc:769] Sending connection context
1026 15:07:34.914786 (+ 100us) client_negotiation.cc:240] Negotiation successful
1026 15:07:34.914900 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20251026 15:07:34.915516 15573 heartbeater.cc:344] Connected to a master server at 127.14.212.126:33997
I20251026 15:07:34.915611 15573 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:34.915805 15573 heartbeater.cc:507] Master 127.14.212.126:33997 requested a full tablet report, sending...
I20251026 15:07:34.916162 15240 ts_manager.cc:194] Registered new tserver with Master: c0ff651bbc7749adb47623a6d7f2afed (127.14.212.66:40799)
I20251026 15:07:34.916707 15240 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:50785
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:34.954435 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33997
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:35.033393 15580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:35.033583 15580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:35.033618 15580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:35.035123 15580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:35.035177 15580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:35.035215 15580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:07:35.036767 15580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:33997
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:35.037000 15580 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:35.037173 15580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:35.039717 15586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:35.039710 15588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:35.039682 15585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:35.039981 15580 server_base.cc:1047] running on GCE node
I20251026 15:07:35.040130 15580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:35.040308 15580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:35.041438 15580 hybrid_clock.cc:648] HybridClock initialized: now 1761491255041424 us; error 26 us; skew 500 ppm
Oct 26 15:07:35 dist-test-slave-8pgz krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491255, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:35.043834 15580 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:07:35.044340 15580 webserver.cc:492] Webserver started at http://127.14.212.67:34965/ using document root <none> and password file <none>
I20251026 15:07:35.044520 15580 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:35.044557 15580 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:35.044665 15580 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:35.045569 15580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "bc0ae080947e4727a388f91e25da4b85"
format_stamp: "Formatted at 2025-10-26 15:07:35 on dist-test-slave-8pgz"
server_key: "f2eed374cae0094c84d53608000705b4"
server_key_iv: "6b629053ab6b3383bda2b1e6036a6f3f"
server_key_version: "encryptionkey@0"
I20251026 15:07:35.045867 15580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "bc0ae080947e4727a388f91e25da4b85"
format_stamp: "Formatted at 2025-10-26 15:07:35 on dist-test-slave-8pgz"
server_key: "f2eed374cae0094c84d53608000705b4"
server_key_iv: "6b629053ab6b3383bda2b1e6036a6f3f"
server_key_version: "encryptionkey@0"
I20251026 15:07:35.047073 15580 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:07:35.047737 15595 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:35.047901 15580 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:35.047976 15580 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "bc0ae080947e4727a388f91e25da4b85"
format_stamp: "Formatted at 2025-10-26 15:07:35 on dist-test-slave-8pgz"
server_key: "f2eed374cae0094c84d53608000705b4"
server_key_iv: "6b629053ab6b3383bda2b1e6036a6f3f"
server_key_version: "encryptionkey@0"
I20251026 15:07:35.048029 15580 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:35.062688 15580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:35.064754 15580 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:35.064879 15580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:35.065090 15580 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:07:35.065372 15580 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:35.065403 15580 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:35.065436 15580 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:35.065464 15580 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:35.071152 15580 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:43431
I20251026 15:07:35.071208 15709 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:43431 every 8 connection(s)
I20251026 15:07:35.071475 15580 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
Oct 26 15:07:35 dist-test-slave-8pgz krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491255, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:35.078823 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 15580
I20251026 15:07:35.078905 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:07:35.079047 15185 external_mini_cluster.cc:1468] Setting key d8c4f95ee0ca2366aeff1c222a2d2f9e
I20251026 15:07:35.079582 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:35.072120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:33905 (local address 127.14.212.126:33997)
1026 15:07:35.072171 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:35.072173 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:35.072300 (+ 127us) server_negotiation.cc:408] Connection header received
1026 15:07:35.072807 (+ 507us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:35.072809 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:35.072818 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:35.072867 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:35.073567 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:35.074045 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:35.074663 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:35.074788 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:35.077024 (+ 2236us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:35.077030 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:35.077030 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:35.077036 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:35.078278 (+ 1242us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:35.078666 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:35.078667 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:35.078668 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:35.078694 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:35.078878 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:35.078879 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:35.078879 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:35.078990 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:35.079054 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:35.079385 (+ 331us) server_negotiation.cc:300] Negotiation successful
1026 15:07:35.079488 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251026 15:07:35.079766 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:35.072090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33997 (local address 127.14.212.67:33905)
1026 15:07:35.072207 (+ 117us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:35.072231 (+ 24us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:35.072739 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:35.072923 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:35.072926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:35.073223 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:35.073503 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:35.073506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:35.074135 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:35.074136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:35.074585 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:35.074587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:35.074707 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:35.075645 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:35.075660 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:35.076946 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:35.078346 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:35.078351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:35.078354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:35.078575 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:35.078754 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:35.078754 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:35.078755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:35.078809 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:35.079043 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:35.079048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:35.079200 (+ 152us) client_negotiation.cc:769] Sending connection context
1026 15:07:35.079298 (+ 98us) client_negotiation.cc:240] Negotiation successful
1026 15:07:35.079443 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":65,"threads_started":1}
I20251026 15:07:35.080279 15710 heartbeater.cc:344] Connected to a master server at 127.14.212.126:33997
I20251026 15:07:35.080366 15710 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:35.080533 15710 heartbeater.cc:507] Master 127.14.212.126:33997 requested a full tablet report, sending...
I20251026 15:07:35.080868 15240 ts_manager.cc:194] Registered new tserver with Master: bc0ae080947e4727a388f91e25da4b85 (127.14.212.67:43431)
I20251026 15:07:35.081238 15240 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:33905
I20251026 15:07:35.091053 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 26 15:07:35 dist-test-slave-8pgz krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491255, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 26 15:07:35 dist-test-slave-8pgz krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491255, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.212.65@KRBTEST.COM
I20251026 15:07:35.111689 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:35.104873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60288 (local address 127.14.212.65:35721)
1026 15:07:35.105014 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:35.105020 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:35.105032 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:07:35.105058 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:35.105060 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:35.105072 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:35.105132 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:35.105769 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:35.106482 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:35.106996 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:35.107158 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:35.109083 (+ 1925us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:35.109092 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:35.109092 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:35.109099 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:35.110724 (+ 1625us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:35.111048 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:35.111050 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:35.111050 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:35.111076 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:35.111186 (+ 110us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:35.111186 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:35.111186 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:35.111267 (+ 81us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:35.111420 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:35.111489 (+ 69us) server_negotiation.cc:300] Negotiation successful
1026 15:07:35.111575 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
W20251026 15:07:35.112035 15348 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60288
Oct 26 15:07:35 dist-test-slave-8pgz krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 26 15:07:35 dist-test-slave-8pgz krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.65@KRBTEST.COM
I20251026 15:07:35.132652 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:35.126105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.14.212.65:35721)
1026 15:07:35.126175 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:35.126178 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:35.126198 (+ 20us) server_negotiation.cc:408] Connection header received
1026 15:07:35.126266 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:35.126268 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:35.126282 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:35.126333 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:35.127030 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:35.127542 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:35.128063 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:35.128198 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:35.130130 (+ 1932us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:35.130134 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:35.130134 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:35.130141 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:35.131605 (+ 1464us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:35.131985 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:35.131986 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:35.131986 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:35.132004 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:35.132181 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:35.132183 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:35.132183 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:35.132270 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:35.132367 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:35.132449 (+ 82us) server_negotiation.cc:300] Negotiation successful
1026 15:07:35.132523 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251026 15:07:35.133198 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 15304
I20251026 15:07:35.138244 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 15441
I20251026 15:07:35.143877 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 15580
I20251026 15:07:35.148406 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 15208
2025-10-26T15:07:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2854 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:07:35 dist-test-slave-8pgz krb5kdc[15724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:07:35 dist-test-slave-8pgz krb5kdc[15724](info): set up 2 sockets
Oct 26 15:07:35 dist-test-slave-8pgz krb5kdc[15724](info): commencing operation
krb5kdc: starting...
W20251026 15:07:37.127408 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.959s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:07:37 dist-test-slave-8pgz krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:07:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:07:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:37.255146 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:44323
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:44323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:37.334985 15740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:37.335166 15740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:37.335189 15740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:37.336549 15740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:07:37.336589 15740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:37.336604 15740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:37.336623 15740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:07:37.336635 15740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:07:37.338238 15740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:44323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:44323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:37.338466 15740 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:37.338636 15740 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:37.341174 15746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:37.341193 15748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:37.341260 15745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:37.341320 15740 server_base.cc:1047] running on GCE node
I20251026 15:07:37.341637 15740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:37.341872 15740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:37.343009 15740 hybrid_clock.cc:648] HybridClock initialized: now 1761491257342981 us; error 52 us; skew 500 ppm
Oct 26 15:07:37 dist-test-slave-8pgz krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491257, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:37.345463 15740 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:07:37.346030 15740 webserver.cc:492] Webserver started at http://127.14.212.126:43199/ using document root <none> and password file <none>
I20251026 15:07:37.346220 15740 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:37.346261 15740 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:37.346374 15740 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:37.347352 15740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "146ce72955484f959a0ed417a318a3f4"
format_stamp: "Formatted at 2025-10-26 15:07:37 on dist-test-slave-8pgz"
server_key: "e8111ca5f96bfafa69298af20c281b27"
server_key_iv: "1fbc19a3cdd6911de3cea55790199ffa"
server_key_version: "encryptionkey@0"
I20251026 15:07:37.347671 15740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "146ce72955484f959a0ed417a318a3f4"
format_stamp: "Formatted at 2025-10-26 15:07:37 on dist-test-slave-8pgz"
server_key: "e8111ca5f96bfafa69298af20c281b27"
server_key_iv: "1fbc19a3cdd6911de3cea55790199ffa"
server_key_version: "encryptionkey@0"
I20251026 15:07:37.348901 15740 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:07:37.349627 15755 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:37.349792 15740 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:07:37.349866 15740 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "146ce72955484f959a0ed417a318a3f4"
format_stamp: "Formatted at 2025-10-26 15:07:37 on dist-test-slave-8pgz"
server_key: "e8111ca5f96bfafa69298af20c281b27"
server_key_iv: "1fbc19a3cdd6911de3cea55790199ffa"
server_key_version: "encryptionkey@0"
I20251026 15:07:37.349929 15740 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:37.360436 15740 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:37.365305 15740 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:37.365463 15740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:37.369400 15740 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:44323
I20251026 15:07:37.369452 15808 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:44323 every 8 connection(s)
I20251026 15:07:37.369756 15740 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:07:37.370334 15809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:37.372762 15809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4: Bootstrap starting.
I20251026 15:07:37.373335 15809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:37.373569 15809 log.cc:826] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:37.374217 15809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4: No bootstrap required, opened a new log
I20251026 15:07:37.375438 15809 raft_consensus.cc:359] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146ce72955484f959a0ed417a318a3f4" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44323 } }
I20251026 15:07:37.375557 15809 raft_consensus.cc:385] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:37.375577 15809 raft_consensus.cc:740] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146ce72955484f959a0ed417a318a3f4, State: Initialized, Role: FOLLOWER
I20251026 15:07:37.375698 15809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146ce72955484f959a0ed417a318a3f4" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44323 } }
I20251026 15:07:37.375761 15809 raft_consensus.cc:399] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:07:37.375787 15809 raft_consensus.cc:493] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:07:37.375830 15809 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:37.376341 15809 raft_consensus.cc:515] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146ce72955484f959a0ed417a318a3f4" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44323 } }
I20251026 15:07:37.376436 15809 leader_election.cc:304] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 146ce72955484f959a0ed417a318a3f4; no voters:
I20251026 15:07:37.376583 15809 leader_election.cc:290] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:07:37.376631 15812 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:37.376791 15809 sys_catalog.cc:565] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:07:37.376797 15812 raft_consensus.cc:697] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [term 1 LEADER]: Becoming Leader. State: Replica: 146ce72955484f959a0ed417a318a3f4, State: Running, Role: LEADER
I20251026 15:07:37.376902 15812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146ce72955484f959a0ed417a318a3f4" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44323 } }
I20251026 15:07:37.377262 15813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "146ce72955484f959a0ed417a318a3f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146ce72955484f959a0ed417a318a3f4" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44323 } } }
I20251026 15:07:37.377296 15815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 146ce72955484f959a0ed417a318a3f4. Latest consensus state: current_term: 1 leader_uuid: "146ce72955484f959a0ed417a318a3f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146ce72955484f959a0ed417a318a3f4" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44323 } } }
I20251026 15:07:37.377434 15813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [sys.catalog]: This master's current role is: LEADER
I20251026 15:07:37.377452 15815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4 [sys.catalog]: This master's current role is: LEADER
I20251026 15:07:37.377838 15823 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:07:37.378329 15823 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:07:37.379528 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 15740
I20251026 15:07:37.379621 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:07:37.379808 15185 external_mini_cluster.cc:1468] Setting key c23b368fd341d0d04303a0d82602310d
I20251026 15:07:37.380088 15823 catalog_manager.cc:1357] Generated new cluster ID: f37ecd2c3b6d4ac9b96f1afffe14cf9f
I20251026 15:07:37.380142 15823 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 26 15:07:37 dist-test-slave-8pgz krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:37.390255 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:37.380423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.14.212.126:44323)
1026 15:07:37.380553 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:37.380564 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:37.380585 (+ 21us) server_negotiation.cc:408] Connection header received
1026 15:07:37.380699 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:37.380704 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:37.380750 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:37.380872 (+ 122us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:37.381780 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.382952 (+ 1172us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:37.383727 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.383960 (+ 233us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:37.386625 (+ 2665us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:37.386632 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:37.386632 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:37.386656 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:37.388798 (+ 2142us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:37.389142 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:37.389145 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:37.389145 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:37.389211 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:37.389360 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:37.389360 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:37.389361 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:37.389627 (+ 266us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:37.389786 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:37.389869 (+ 83us) server_negotiation.cc:300] Negotiation successful
1026 15:07:37.390024 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20251026 15:07:37.392228 15823 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:07:37.392740 15823 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:07:37.403118 15823 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 146ce72955484f959a0ed417a318a3f4: Generated new TSK 0
I20251026 15:07:37.403321 15823 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:37.461678 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:44323
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:37.561335 15839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:37.561498 15839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:37.561522 15839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:37.563084 15839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:37.563136 15839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:37.563165 15839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:07:37.564791 15839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:44323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:37.565006 15839 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:37.565191 15839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:37.567790 15846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:37.567790 15845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:37.567870 15848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:37.568243 15839 server_base.cc:1047] running on GCE node
I20251026 15:07:37.568414 15839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:37.568610 15839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:37.569747 15839 hybrid_clock.cc:648] HybridClock initialized: now 1761491257569730 us; error 43 us; skew 500 ppm
Oct 26 15:07:37 dist-test-slave-8pgz krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491257, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:37.572189 15839 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:07:37.572744 15839 webserver.cc:492] Webserver started at http://127.14.212.65:45249/ using document root <none> and password file <none>
I20251026 15:07:37.572944 15839 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:37.572985 15839 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:37.573101 15839 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:37.574045 15839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "cb258cf6e050472aa7075a17c7a96b87"
format_stamp: "Formatted at 2025-10-26 15:07:37 on dist-test-slave-8pgz"
server_key: "ac756276563477df1271308771de23f5"
server_key_iv: "d453d070844c0144618fc4a238fcc502"
server_key_version: "encryptionkey@0"
I20251026 15:07:37.574342 15839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "cb258cf6e050472aa7075a17c7a96b87"
format_stamp: "Formatted at 2025-10-26 15:07:37 on dist-test-slave-8pgz"
server_key: "ac756276563477df1271308771de23f5"
server_key_iv: "d453d070844c0144618fc4a238fcc502"
server_key_version: "encryptionkey@0"
I20251026 15:07:37.575531 15839 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:37.576175 15855 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:37.576340 15839 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:37.576426 15839 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "cb258cf6e050472aa7075a17c7a96b87"
format_stamp: "Formatted at 2025-10-26 15:07:37 on dist-test-slave-8pgz"
server_key: "ac756276563477df1271308771de23f5"
server_key_iv: "d453d070844c0144618fc4a238fcc502"
server_key_version: "encryptionkey@0"
I20251026 15:07:37.576476 15839 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:37.586474 15839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:37.588241 15839 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:37.588341 15839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:37.588505 15839 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:07:37.588745 15839 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:37.588774 15839 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:37.588796 15839 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:37.588814 15839 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:37.594466 15839 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:41819
I20251026 15:07:37.594470 15970 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:41819 every 8 connection(s)
I20251026 15:07:37.594805 15839 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:07:37.596738 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 15839
I20251026 15:07:37.596815 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:07:37.596961 15185 external_mini_cluster.cc:1468] Setting key 865f485c7c1e5df5385b1aad5bf409df
Oct 26 15:07:37 dist-test-slave-8pgz krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491257, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:37.603902 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:37.595434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:44599 (local address 127.14.212.126:44323)
1026 15:07:37.595484 (+ 50us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:37.595487 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:37.595632 (+ 145us) server_negotiation.cc:408] Connection header received
1026 15:07:37.596142 (+ 510us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:37.596143 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:37.596159 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:37.596220 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:37.597028 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.597729 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:37.598846 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.598998 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:37.601097 (+ 2099us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:37.601100 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:37.601101 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:37.601110 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:37.602734 (+ 1624us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:37.603111 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:37.603112 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:37.603113 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:37.603138 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:37.603306 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:37.603307 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:37.603307 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:37.603403 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:37.603464 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:37.603712 (+ 248us) server_negotiation.cc:300] Negotiation successful
1026 15:07:37.603808 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
I20251026 15:07:37.604063 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:37.595371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:44323 (local address 127.14.212.65:44599)
1026 15:07:37.595525 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:37.595555 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:37.596047 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:37.596283 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:37.596287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:37.596610 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:37.596941 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:37.596944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.597801 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:37.597802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:37.598724 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:37.598728 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.598883 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:37.599416 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:37.599442 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:37.601006 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:37.602820 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:37.602827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:37.602830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:37.603043 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:37.603193 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:37.603193 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:37.603194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:37.603253 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:37.603459 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:37.603464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:37.603587 (+ 123us) client_negotiation.cc:769] Sending connection context
1026 15:07:37.603692 (+ 105us) client_negotiation.cc:240] Negotiation successful
1026 15:07:37.603824 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20251026 15:07:37.604514 15971 heartbeater.cc:344] Connected to a master server at 127.14.212.126:44323
I20251026 15:07:37.604610 15971 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:37.604820 15971 heartbeater.cc:507] Master 127.14.212.126:44323 requested a full tablet report, sending...
I20251026 15:07:37.605301 15772 ts_manager.cc:194] Registered new tserver with Master: cb258cf6e050472aa7075a17c7a96b87 (127.14.212.65:41819)
I20251026 15:07:37.605896 15772 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:44599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:37.640638 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:44323
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:37.719735 15978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:37.719913 15978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:37.719949 15978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:37.721341 15978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:37.721387 15978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:37.721426 15978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:07:37.722990 15978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:44323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:37.723253 15978 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:37.723481 15978 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:37.726086 15986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:37.726009 15983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:37.726014 15984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:37.726341 15978 server_base.cc:1047] running on GCE node
I20251026 15:07:37.726526 15978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:37.726747 15978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:37.727875 15978 hybrid_clock.cc:648] HybridClock initialized: now 1761491257727853 us; error 37 us; skew 500 ppm
Oct 26 15:07:37 dist-test-slave-8pgz krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491257, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:37.730119 15978 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:07:37.730659 15978 webserver.cc:492] Webserver started at http://127.14.212.66:34463/ using document root <none> and password file <none>
I20251026 15:07:37.730948 15978 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:37.730995 15978 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:37.731107 15978 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:37.732048 15978 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "e624f04812ef4d43816aa9c5d9b74620"
format_stamp: "Formatted at 2025-10-26 15:07:37 on dist-test-slave-8pgz"
server_key: "fb99a4fad83e627aabb23790fbdb889b"
server_key_iv: "35902f9d863d69ad4b8122faef42ea86"
server_key_version: "encryptionkey@0"
I20251026 15:07:37.732331 15978 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "e624f04812ef4d43816aa9c5d9b74620"
format_stamp: "Formatted at 2025-10-26 15:07:37 on dist-test-slave-8pgz"
server_key: "fb99a4fad83e627aabb23790fbdb889b"
server_key_iv: "35902f9d863d69ad4b8122faef42ea86"
server_key_version: "encryptionkey@0"
I20251026 15:07:37.733547 15978 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251026 15:07:37.734237 15993 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:37.734467 15978 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:37.734539 15978 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "e624f04812ef4d43816aa9c5d9b74620"
format_stamp: "Formatted at 2025-10-26 15:07:37 on dist-test-slave-8pgz"
server_key: "fb99a4fad83e627aabb23790fbdb889b"
server_key_iv: "35902f9d863d69ad4b8122faef42ea86"
server_key_version: "encryptionkey@0"
I20251026 15:07:37.734613 15978 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:37.776808 15978 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:37.779232 15978 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:37.779335 15978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:37.779544 15978 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:07:37.779851 15978 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:37.779882 15978 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:37.779937 15978 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:37.779959 15978 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:37.785647 15978 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:41673
I20251026 15:07:37.785709 16106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:41673 every 8 connection(s)
I20251026 15:07:37.785969 15978 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
Oct 26 15:07:37 dist-test-slave-8pgz krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491257, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:37.793956 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:37.786614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:43937 (local address 127.14.212.126:44323)
1026 15:07:37.786682 (+ 68us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:37.786685 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:37.786828 (+ 143us) server_negotiation.cc:408] Connection header received
1026 15:07:37.787326 (+ 498us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:37.787327 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:37.787341 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:37.787403 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:37.788150 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.788649 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:37.789636 (+ 987us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.789836 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:37.791665 (+ 1829us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:37.791669 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:37.791669 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:37.791675 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:37.792867 (+ 1192us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:37.793206 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:37.793207 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:37.793207 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:37.793236 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:37.793396 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:37.793396 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:37.793397 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:37.793484 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:37.793543 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:37.793778 (+ 235us) server_negotiation.cc:300] Negotiation successful
1026 15:07:37.793871 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251026 15:07:37.794095 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:37.786584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:44323 (local address 127.14.212.66:43937)
1026 15:07:37.786729 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:37.786756 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:37.787238 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:37.787463 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:37.787466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:37.787773 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:37.788058 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:37.788060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.788732 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:37.788732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:37.789556 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:37.789558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.789687 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:37.790173 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:37.790188 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:37.791573 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:37.792945 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:37.792951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:37.792953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:37.793140 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:37.793287 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:37.793288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:37.793288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:37.793344 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:37.793536 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:37.793540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:37.793662 (+ 122us) client_negotiation.cc:769] Sending connection context
1026 15:07:37.793756 (+ 94us) client_negotiation.cc:240] Negotiation successful
1026 15:07:37.793877 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20251026 15:07:37.794549 16107 heartbeater.cc:344] Connected to a master server at 127.14.212.126:44323
I20251026 15:07:37.794641 16107 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:37.794862 16107 heartbeater.cc:507] Master 127.14.212.126:44323 requested a full tablet report, sending...
I20251026 15:07:37.795208 15772 ts_manager.cc:194] Registered new tserver with Master: e624f04812ef4d43816aa9c5d9b74620 (127.14.212.66:41673)
I20251026 15:07:37.795469 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 15978
I20251026 15:07:37.795552 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:07:37.795584 15772 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:43937
I20251026 15:07:37.795830 15185 external_mini_cluster.cc:1468] Setting key d1b38ed0f214485081981dbad1f1a2b1
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:37.840808 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:44323
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:37.919718 16115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:37.919880 16115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:37.919903 16115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:37.921270 16115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:37.921316 16115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:37.921342 16115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:07:37.922904 16115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:44323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:37.923110 16115 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:37.923288 16115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:37.925772 16121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:37.925784 16123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:37.925783 16120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:37.925968 16115 server_base.cc:1047] running on GCE node
I20251026 15:07:37.926182 16115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:37.926373 16115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:37.927542 16115 hybrid_clock.cc:648] HybridClock initialized: now 1761491257927532 us; error 30 us; skew 500 ppm
Oct 26 15:07:37 dist-test-slave-8pgz krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491257, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:37.929791 16115 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:07:37.930289 16115 webserver.cc:492] Webserver started at http://127.14.212.67:36539/ using document root <none> and password file <none>
I20251026 15:07:37.930502 16115 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:37.930544 16115 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:37.930655 16115 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:37.931603 16115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "da5082ab45194afabe82b745dc5e2115"
format_stamp: "Formatted at 2025-10-26 15:07:37 on dist-test-slave-8pgz"
server_key: "566b67f4d78ba0530cf34b2dd4dcb1a8"
server_key_iv: "2f532b107820813208649b0b01e53778"
server_key_version: "encryptionkey@0"
I20251026 15:07:37.931895 16115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "da5082ab45194afabe82b745dc5e2115"
format_stamp: "Formatted at 2025-10-26 15:07:37 on dist-test-slave-8pgz"
server_key: "566b67f4d78ba0530cf34b2dd4dcb1a8"
server_key_iv: "2f532b107820813208649b0b01e53778"
server_key_version: "encryptionkey@0"
I20251026 15:07:37.933056 16115 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:07:37.933678 16130 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:37.933842 16115 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:37.933919 16115 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "da5082ab45194afabe82b745dc5e2115"
format_stamp: "Formatted at 2025-10-26 15:07:37 on dist-test-slave-8pgz"
server_key: "566b67f4d78ba0530cf34b2dd4dcb1a8"
server_key_iv: "2f532b107820813208649b0b01e53778"
server_key_version: "encryptionkey@0"
I20251026 15:07:37.933977 16115 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:37.963836 16115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:37.965894 16115 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:37.966042 16115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:37.966276 16115 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:07:37.966660 16115 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:37.966696 16115 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:37.966734 16115 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:37.966765 16115 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:37.972391 16115 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:41087
I20251026 15:07:37.972455 16248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:41087 every 8 connection(s)
I20251026 15:07:37.972754 16115 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:07:37.975199 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 16115
I20251026 15:07:37.975289 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:07:37.975425 15185 external_mini_cluster.cc:1468] Setting key 7c414ddefda18a7926d96107fef69b82
Oct 26 15:07:37 dist-test-slave-8pgz krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491257, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:37.981282 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:37.973423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:48535 (local address 127.14.212.126:44323)
1026 15:07:37.973478 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:37.973480 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:37.973695 (+ 215us) server_negotiation.cc:408] Connection header received
1026 15:07:37.974184 (+ 489us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:37.974186 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:37.974198 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:37.974257 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:37.975132 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.975836 (+ 704us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:37.976746 (+ 910us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.976883 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:37.978896 (+ 2013us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:37.978900 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:37.978900 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:37.978907 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:37.980337 (+ 1430us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:37.980666 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:37.980666 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:37.980667 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:37.980683 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:37.980814 (+ 131us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:37.980815 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:37.980815 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:37.980897 (+ 82us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:37.980942 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:37.981150 (+ 208us) server_negotiation.cc:300] Negotiation successful
1026 15:07:37.981217 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251026 15:07:37.981386 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:37.973380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:44323 (local address 127.14.212.67:48535)
1026 15:07:37.973523 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:37.973553 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:37.974079 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:37.974313 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:37.974316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:37.974729 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:37.975025 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:37.975029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.975914 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:37.975915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:37.976656 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:37.976659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.976795 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:37.977312 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:37.977337 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:37.978820 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:37.980414 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:37.980419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:37.980421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:37.980611 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:37.980726 (+ 115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:37.980726 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:37.980727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:37.980776 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:37.980939 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:37.980942 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:37.981050 (+ 108us) client_negotiation.cc:769] Sending connection context
1026 15:07:37.981131 (+ 81us) client_negotiation.cc:240] Negotiation successful
1026 15:07:37.981229 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20251026 15:07:37.981907 16249 heartbeater.cc:344] Connected to a master server at 127.14.212.126:44323
I20251026 15:07:37.982008 16249 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:37.982216 16249 heartbeater.cc:507] Master 127.14.212.126:44323 requested a full tablet report, sending...
I20251026 15:07:37.982580 15772 ts_manager.cc:194] Registered new tserver with Master: da5082ab45194afabe82b745dc5e2115 (127.14.212.67:41087)
I20251026 15:07:37.982956 15772 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:48535
I20251026 15:07:37.987303 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:07:37.994251 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:37.988744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.14.212.126:44323)
1026 15:07:37.988796 (+ 52us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:37.988797 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:37.988940 (+ 143us) server_negotiation.cc:408] Connection header received
1026 15:07:37.989006 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:37.989007 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:37.989015 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:37.989064 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:37.989692 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.990160 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:37.990779 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:37.990887 (+ 108us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:37.992021 (+ 1134us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:37.992025 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:37.992025 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:37.992031 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:37.993376 (+ 1345us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:37.993725 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:37.993726 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:37.993726 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:37.993744 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:37.993854 (+ 110us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:37.993855 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:37.993855 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:37.993957 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:37.994041 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:37.994123 (+ 82us) server_negotiation.cc:300] Negotiation successful
1026 15:07:37.994185 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251026 15:07:37.996272 15772 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:07:37.997517 15772 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:07:38.002657 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:37.999765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:41673 (local address 127.0.0.1:45424)
1026 15:07:37.999932 (+ 167us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:37.999960 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:38.000038 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:38.000338 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:38.000340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:38.000343 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:38.000621 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:38.000623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.001625 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:38.001626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:38.002332 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:38.002334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.002432 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:38.002468 (+ 36us) client_negotiation.cc:769] Sending connection context
1026 15:07:38.002518 (+ 50us) client_negotiation.cc:240] Negotiation successful
1026 15:07:38.002557 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20251026 15:07:38.003197 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:38.000012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.14.212.66:41673)
1026 15:07:38.000182 (+ 170us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:38.000190 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:38.000200 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:07:38.000239 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:38.000242 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:38.000260 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:38.000313 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:38.000714 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.001547 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:38.002462 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.002982 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:38.003037 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:38.003069 (+ 32us) server_negotiation.cc:300] Negotiation successful
1026 15:07:38.003119 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20251026 15:07:38.003255 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:37.999765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:41819 (local address 127.0.0.1:52244)
1026 15:07:37.999935 (+ 170us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:37.999960 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:38.000036 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:38.000594 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:38.000595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:38.000597 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:38.000789 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:38.000790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.002098 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:38.002099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:38.003027 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:38.003030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.003094 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:38.003097 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:07:38.003131 (+ 34us) client_negotiation.cc:240] Negotiation successful
1026 15:07:38.003171 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20251026 15:07:38.003803 16041 tablet_service.cc:1505] Processing CreateTablet for tablet e1aaf328b40b4592870f2a4037212747 (DEFAULT_TABLE table=test-table [id=d85031877f19473ab6c187b051eed3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:38.003950 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:37.999961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.14.212.65:41819)
1026 15:07:38.000426 (+ 465us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:38.000434 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:38.000443 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:07:38.000480 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:38.000482 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:38.000496 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:38.000569 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:38.000880 (+ 311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.002011 (+ 1131us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:38.003384 (+ 1373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.003813 (+ 429us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:38.003849 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:38.003872 (+ 23us) server_negotiation.cc:300] Negotiation successful
1026 15:07:38.003899 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":82,"threads_started":1}
I20251026 15:07:38.004132 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:38.000120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.14.212.67:41087)
1026 15:07:38.000292 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:38.000300 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:38.000309 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:07:38.000347 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:38.000349 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:38.000367 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:38.000438 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:38.000772 (+ 334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.002020 (+ 1248us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:38.003423 (+ 1403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.003927 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:38.003977 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:38.004008 (+ 31us) server_negotiation.cc:300] Negotiation successful
1026 15:07:38.004054 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20251026 15:07:38.004140 16041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1aaf328b40b4592870f2a4037212747. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:38.004256 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:37.999774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:41087 (local address 127.0.0.1:40954)
1026 15:07:38.000160 (+ 386us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:38.000168 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:38.000205 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:38.000461 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:38.000463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:38.000465 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:38.000679 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:38.000682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.002133 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:38.002134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:38.003044 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:38.003047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.003119 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:38.003122 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:07:38.003154 (+ 32us) client_negotiation.cc:240] Negotiation successful
1026 15:07:38.003191 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":89,"threads_started":1}
I20251026 15:07:38.004796 15904 tablet_service.cc:1505] Processing CreateTablet for tablet e1aaf328b40b4592870f2a4037212747 (DEFAULT_TABLE table=test-table [id=d85031877f19473ab6c187b051eed3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:38.004907 16181 tablet_service.cc:1505] Processing CreateTablet for tablet e1aaf328b40b4592870f2a4037212747 (DEFAULT_TABLE table=test-table [id=d85031877f19473ab6c187b051eed3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:38.005077 15904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1aaf328b40b4592870f2a4037212747. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:38.005151 16181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1aaf328b40b4592870f2a4037212747. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:38.006412 16267 tablet_bootstrap.cc:492] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620: Bootstrap starting.
I20251026 15:07:38.007150 16267 tablet_bootstrap.cc:654] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:38.007369 16267 log.cc:826] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:38.007812 16269 tablet_bootstrap.cc:492] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87: Bootstrap starting.
I20251026 15:07:38.008119 16267 tablet_bootstrap.cc:492] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620: No bootstrap required, opened a new log
I20251026 15:07:38.008185 16267 ts_tablet_manager.cc:1403] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:07:38.008406 16269 tablet_bootstrap.cc:654] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:38.008702 16269 log.cc:826] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:38.008988 16270 tablet_bootstrap.cc:492] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115: Bootstrap starting.
I20251026 15:07:38.009296 16269 tablet_bootstrap.cc:492] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87: No bootstrap required, opened a new log
I20251026 15:07:38.009356 16269 ts_tablet_manager.cc:1403] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:07:38.009545 16270 tablet_bootstrap.cc:654] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:38.009853 16270 log.cc:826] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:38.010141 16267 raft_consensus.cc:359] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb258cf6e050472aa7075a17c7a96b87" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41819 } } peers { permanent_uuid: "e624f04812ef4d43816aa9c5d9b74620" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 41673 } } peers { permanent_uuid: "da5082ab45194afabe82b745dc5e2115" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41087 } }
I20251026 15:07:38.010284 16267 raft_consensus.cc:385] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:38.010319 16267 raft_consensus.cc:740] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e624f04812ef4d43816aa9c5d9b74620, State: Initialized, Role: FOLLOWER
I20251026 15:07:38.010455 16267 consensus_queue.cc:260] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb258cf6e050472aa7075a17c7a96b87" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41819 } } peers { permanent_uuid: "e624f04812ef4d43816aa9c5d9b74620" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 41673 } } peers { permanent_uuid: "da5082ab45194afabe82b745dc5e2115" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41087 } }
I20251026 15:07:38.010712 16267 ts_tablet_manager.cc:1434] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251026 15:07:38.010776 16270 tablet_bootstrap.cc:492] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115: No bootstrap required, opened a new log
I20251026 15:07:38.010821 16107 heartbeater.cc:499] Master 127.14.212.126:44323 was elected leader, sending a full tablet report...
I20251026 15:07:38.010828 16270 ts_tablet_manager.cc:1403] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:07:38.011056 16269 raft_consensus.cc:359] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb258cf6e050472aa7075a17c7a96b87" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41819 } } peers { permanent_uuid: "e624f04812ef4d43816aa9c5d9b74620" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 41673 } } peers { permanent_uuid: "da5082ab45194afabe82b745dc5e2115" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41087 } }
I20251026 15:07:38.011189 16269 raft_consensus.cc:385] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:38.011217 16269 raft_consensus.cc:740] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb258cf6e050472aa7075a17c7a96b87, State: Initialized, Role: FOLLOWER
I20251026 15:07:38.011299 16269 consensus_queue.cc:260] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb258cf6e050472aa7075a17c7a96b87" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41819 } } peers { permanent_uuid: "e624f04812ef4d43816aa9c5d9b74620" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 41673 } } peers { permanent_uuid: "da5082ab45194afabe82b745dc5e2115" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41087 } }
I20251026 15:07:38.011534 16269 ts_tablet_manager.cc:1434] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:07:38.011658 15971 heartbeater.cc:499] Master 127.14.212.126:44323 was elected leader, sending a full tablet report...
I20251026 15:07:38.012394 16270 raft_consensus.cc:359] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb258cf6e050472aa7075a17c7a96b87" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41819 } } peers { permanent_uuid: "e624f04812ef4d43816aa9c5d9b74620" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 41673 } } peers { permanent_uuid: "da5082ab45194afabe82b745dc5e2115" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41087 } }
I20251026 15:07:38.012523 16270 raft_consensus.cc:385] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:38.012562 16270 raft_consensus.cc:740] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da5082ab45194afabe82b745dc5e2115, State: Initialized, Role: FOLLOWER
I20251026 15:07:38.012653 16270 consensus_queue.cc:260] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb258cf6e050472aa7075a17c7a96b87" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41819 } } peers { permanent_uuid: "e624f04812ef4d43816aa9c5d9b74620" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 41673 } } peers { permanent_uuid: "da5082ab45194afabe82b745dc5e2115" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41087 } }
I20251026 15:07:38.012902 16249 heartbeater.cc:499] Master 127.14.212.126:44323 was elected leader, sending a full tablet report...
I20251026 15:07:38.012995 16270 ts_tablet_manager.cc:1434] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251026 15:07:38.036443 16108 tablet.cc:2378] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 15:07:38.095522 15972 tablet.cc:2378] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:07:38.123013 16276 raft_consensus.cc:493] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:07:38.123132 16276 raft_consensus.cc:515] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb258cf6e050472aa7075a17c7a96b87" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41819 } } peers { permanent_uuid: "e624f04812ef4d43816aa9c5d9b74620" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 41673 } } peers { permanent_uuid: "da5082ab45194afabe82b745dc5e2115" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41087 } }
I20251026 15:07:38.123447 16276 leader_election.cc:290] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb258cf6e050472aa7075a17c7a96b87 (127.14.212.65:41819), e624f04812ef4d43816aa9c5d9b74620 (127.14.212.66:41673)
I20251026 15:07:38.125600 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:38.123624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:41819 (local address 127.14.212.67:48881)
1026 15:07:38.123684 (+ 60us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:38.123696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:38.123740 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:38.123946 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:38.123947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:38.123949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:38.124124 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:38.124125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.124754 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:38.124755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:38.125386 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:38.125388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.125457 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:38.125461 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:07:38.125493 (+ 32us) client_negotiation.cc:240] Negotiation successful
1026 15:07:38.125531 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20251026 15:07:38.125986 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:38.123636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:41673 (local address 127.14.212.67:35799)
1026 15:07:38.123832 (+ 196us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:38.123841 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:38.123943 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:38.124093 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:38.124094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:38.124095 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:38.124321 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:38.124322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.125110 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:38.125111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:38.125702 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:38.125705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.125835 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:38.125838 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:07:38.125870 (+ 32us) client_negotiation.cc:240] Negotiation successful
1026 15:07:38.125920 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":110,"threads_started":1}
I20251026 15:07:38.126055 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:38.123782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:48881 (local address 127.14.212.65:41819)
1026 15:07:38.123825 (+ 43us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:38.123827 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:38.123834 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:07:38.123858 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:38.123859 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:38.123870 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:38.123921 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:38.124197 (+ 276us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.124670 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:38.125468 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.125931 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:38.125941 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:38.125962 (+ 21us) server_negotiation.cc:300] Negotiation successful
1026 15:07:38.125993 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251026 15:07:38.126353 15924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1aaf328b40b4592870f2a4037212747" candidate_uuid: "da5082ab45194afabe82b745dc5e2115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb258cf6e050472aa7075a17c7a96b87" is_pre_election: true
I20251026 15:07:38.126464 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:38.123736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:35799 (local address 127.14.212.66:41673)
1026 15:07:38.123802 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:38.123805 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:38.123906 (+ 101us) server_negotiation.cc:408] Connection header received
1026 15:07:38.124010 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:38.124011 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:38.124023 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:38.124079 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:38.124398 (+ 319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.125012 (+ 614us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:38.125859 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.126307 (+ 448us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:38.126314 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:38.126332 (+ 18us) server_negotiation.cc:300] Negotiation successful
1026 15:07:38.126375 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251026 15:07:38.126557 15924 raft_consensus.cc:2468] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da5082ab45194afabe82b745dc5e2115 in term 0.
I20251026 15:07:38.126762 16061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1aaf328b40b4592870f2a4037212747" candidate_uuid: "da5082ab45194afabe82b745dc5e2115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e624f04812ef4d43816aa9c5d9b74620" is_pre_election: true
I20251026 15:07:38.126828 16132 leader_election.cc:304] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb258cf6e050472aa7075a17c7a96b87, da5082ab45194afabe82b745dc5e2115; no voters:
I20251026 15:07:38.126902 16061 raft_consensus.cc:2468] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da5082ab45194afabe82b745dc5e2115 in term 0.
I20251026 15:07:38.126945 16276 raft_consensus.cc:2804] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:07:38.127007 16276 raft_consensus.cc:493] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:07:38.127032 16276 raft_consensus.cc:3060] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:38.127722 16276 raft_consensus.cc:515] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb258cf6e050472aa7075a17c7a96b87" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41819 } } peers { permanent_uuid: "e624f04812ef4d43816aa9c5d9b74620" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 41673 } } peers { permanent_uuid: "da5082ab45194afabe82b745dc5e2115" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41087 } }
I20251026 15:07:38.127912 16276 leader_election.cc:290] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [CANDIDATE]: Term 1 election: Requested vote from peers cb258cf6e050472aa7075a17c7a96b87 (127.14.212.65:41819), e624f04812ef4d43816aa9c5d9b74620 (127.14.212.66:41673)
I20251026 15:07:38.128001 16061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1aaf328b40b4592870f2a4037212747" candidate_uuid: "da5082ab45194afabe82b745dc5e2115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e624f04812ef4d43816aa9c5d9b74620"
I20251026 15:07:38.128069 16061 raft_consensus.cc:3060] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:38.128698 16061 raft_consensus.cc:2468] T e1aaf328b40b4592870f2a4037212747 P e624f04812ef4d43816aa9c5d9b74620 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da5082ab45194afabe82b745dc5e2115 in term 1.
I20251026 15:07:38.128794 15924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1aaf328b40b4592870f2a4037212747" candidate_uuid: "da5082ab45194afabe82b745dc5e2115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb258cf6e050472aa7075a17c7a96b87"
I20251026 15:07:38.128880 15924 raft_consensus.cc:3060] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:38.128901 16134 leader_election.cc:304] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da5082ab45194afabe82b745dc5e2115, e624f04812ef4d43816aa9c5d9b74620; no voters:
I20251026 15:07:38.128996 16276 raft_consensus.cc:2804] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:38.129146 16276 raft_consensus.cc:697] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [term 1 LEADER]: Becoming Leader. State: Replica: da5082ab45194afabe82b745dc5e2115, State: Running, Role: LEADER
I20251026 15:07:38.129249 16276 consensus_queue.cc:237] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb258cf6e050472aa7075a17c7a96b87" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41819 } } peers { permanent_uuid: "e624f04812ef4d43816aa9c5d9b74620" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 41673 } } peers { permanent_uuid: "da5082ab45194afabe82b745dc5e2115" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41087 } }
I20251026 15:07:38.129798 15924 raft_consensus.cc:2468] T e1aaf328b40b4592870f2a4037212747 P cb258cf6e050472aa7075a17c7a96b87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da5082ab45194afabe82b745dc5e2115 in term 1.
I20251026 15:07:38.130064 15771 catalog_manager.cc:5649] T e1aaf328b40b4592870f2a4037212747 P da5082ab45194afabe82b745dc5e2115 reported cstate change: term changed from 0 to 1, leader changed from <none> to da5082ab45194afabe82b745dc5e2115 (127.14.212.67). New cstate: current_term: 1 leader_uuid: "da5082ab45194afabe82b745dc5e2115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb258cf6e050472aa7075a17c7a96b87" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e624f04812ef4d43816aa9c5d9b74620" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da5082ab45194afabe82b745dc5e2115" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41087 } health_report { overall_health: HEALTHY } } }
Oct 26 15:07:38 dist-test-slave-8pgz krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.65@KRBTEST.COM
I20251026 15:07:38.155906 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:38.148519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.14.212.65:41819)
1026 15:07:38.148608 (+ 89us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:38.148611 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:38.148688 (+ 77us) server_negotiation.cc:408] Connection header received
1026 15:07:38.148787 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:38.148788 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:38.148801 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:38.148851 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:38.149549 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.150007 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:38.150621 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.150814 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:38.152911 (+ 2097us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:38.152924 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:38.152924 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:38.152930 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:38.154796 (+ 1866us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:38.155120 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:38.155123 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:38.155123 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:38.155158 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:38.155310 (+ 152us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:38.155310 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:38.155311 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:38.155408 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:38.155564 (+ 156us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:38.155601 (+ 37us) server_negotiation.cc:300] Negotiation successful
1026 15:07:38.155696 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Oct 26 15:07:38 dist-test-slave-8pgz krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491258, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 26 15:07:38 dist-test-slave-8pgz krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491258, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.212.65@KRBTEST.COM
I20251026 15:07:38.177448 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:38.171125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52268 (local address 127.14.212.65:41819)
1026 15:07:38.171172 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:38.171174 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:38.171261 (+ 87us) server_negotiation.cc:408] Connection header received
1026 15:07:38.171317 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:38.171318 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:38.171327 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:38.171376 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:38.172059 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.172545 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:38.173124 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.173256 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:38.175341 (+ 2085us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:38.175345 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:38.175345 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:38.175351 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:38.176636 (+ 1285us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:38.176917 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:38.176918 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:38.176918 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:38.176943 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:38.177047 (+ 104us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:38.177047 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:38.177048 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:38.177135 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:38.177204 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:38.177308 (+ 104us) server_negotiation.cc:300] Negotiation successful
1026 15:07:38.177369 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Oct 26 15:07:38 dist-test-slave-8pgz krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 26 15:07:38 dist-test-slave-8pgz krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.65@KRBTEST.COM
I20251026 15:07:38.198145 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:38.191915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.14.212.65:41819)
1026 15:07:38.191994 (+ 79us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:38.191997 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:38.192068 (+ 71us) server_negotiation.cc:408] Connection header received
1026 15:07:38.192144 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:38.192145 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:38.192156 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:38.192209 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:38.192886 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.193341 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:38.193894 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:38.194091 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:38.195996 (+ 1905us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:38.196003 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:38.196003 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:38.196009 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:38.197238 (+ 1229us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:38.197526 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:38.197527 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:38.197528 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:38.197550 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:38.197694 (+ 144us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:38.197694 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:38.197695 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:38.197769 (+ 74us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:38.197855 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:38.197965 (+ 110us) server_negotiation.cc:300] Negotiation successful
1026 15:07:38.198043 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20251026 15:07:38.198580 15884 tablet_service.cc:3012] Rejecting scan request for tablet e1aaf328b40b4592870f2a4037212747: Uninitialized: clean time has not yet been initialized
I20251026 15:07:38.199285 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 15839
I20251026 15:07:38.204500 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 15978
I20251026 15:07:38.208992 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 16115
I20251026 15:07:38.213348 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 15740
2025-10-26T15:07:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3066 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:07:38 dist-test-slave-8pgz krb5kdc[16292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:07:38 dist-test-slave-8pgz krb5kdc[16292](info): set up 2 sockets
Oct 26 15:07:38 dist-test-slave-8pgz krb5kdc[16292](info): commencing operation
krb5kdc: starting...
W20251026 15:07:40.203600 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.969s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:07:40 dist-test-slave-8pgz krb5kdc[16292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:07:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:07:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:40.338128 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:36573
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:36573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:40.417466 16309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:40.417691 16309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:40.417728 16309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:40.419188 16309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:07:40.419235 16309 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251026 15:07:40.419255 16309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:40.419279 16309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:40.419297 16309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:07:40.419317 16309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:07:40.420964 16309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:36573
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:36573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:40.421221 16309 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:40.421407 16309 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:40.423954 16317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:40.424034 16309 server_base.cc:1047] running on GCE node
W20251026 15:07:40.423954 16314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:40.423978 16315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:40.424396 16309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:40.424611 16309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:40.425763 16309 hybrid_clock.cc:648] HybridClock initialized: now 1761491260425757 us; error 50 us; skew 500 ppm
Oct 26 15:07:40 dist-test-slave-8pgz krb5kdc[16292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491260, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:40.428145 16309 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:07:40.428685 16309 webserver.cc:492] Webserver started at http://127.14.212.126:43681/ using document root <none> and password file <none>
I20251026 15:07:40.428869 16309 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:40.428910 16309 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:40.429021 16309 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:40.429970 16309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "7088cfbe1e3e4e708cc7fd8afa3463ca"
format_stamp: "Formatted at 2025-10-26 15:07:40 on dist-test-slave-8pgz"
server_key: "ab611c3e59ceb399c3ceaf3f89395872"
server_key_iv: "01824e849107c1a82246f9bbcf811163"
server_key_version: "encryptionkey@0"
I20251026 15:07:40.430274 16309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "7088cfbe1e3e4e708cc7fd8afa3463ca"
format_stamp: "Formatted at 2025-10-26 15:07:40 on dist-test-slave-8pgz"
server_key: "ab611c3e59ceb399c3ceaf3f89395872"
server_key_iv: "01824e849107c1a82246f9bbcf811163"
server_key_version: "encryptionkey@0"
I20251026 15:07:40.431494 16309 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:07:40.432193 16325 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:40.432394 16309 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:07:40.432474 16309 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "7088cfbe1e3e4e708cc7fd8afa3463ca"
format_stamp: "Formatted at 2025-10-26 15:07:40 on dist-test-slave-8pgz"
server_key: "ab611c3e59ceb399c3ceaf3f89395872"
server_key_iv: "01824e849107c1a82246f9bbcf811163"
server_key_version: "encryptionkey@0"
I20251026 15:07:40.432543 16309 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:40.440181 16309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:40.441704 16309 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:40.441814 16309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:40.446211 16309 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:36573
I20251026 15:07:40.446251 16389 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:36573 every 8 connection(s)
I20251026 15:07:40.446590 16309 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:07:40.447158 16390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:40.449626 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca: Bootstrap starting.
I20251026 15:07:40.450150 16390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:40.450359 16390 log.cc:826] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:40.451052 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca: No bootstrap required, opened a new log
I20251026 15:07:40.452163 16390 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7088cfbe1e3e4e708cc7fd8afa3463ca" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 36573 } }
I20251026 15:07:40.452327 16390 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:40.452363 16390 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7088cfbe1e3e4e708cc7fd8afa3463ca, State: Initialized, Role: FOLLOWER
I20251026 15:07:40.452435 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 16309
I20251026 15:07:40.452517 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:07:40.452497 16390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7088cfbe1e3e4e708cc7fd8afa3463ca" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 36573 } }
I20251026 15:07:40.452564 16390 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:07:40.452606 16390 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:07:40.452643 16390 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:40.452667 15185 external_mini_cluster.cc:1468] Setting key 814b361473e499b3e9e48515a3137258
I20251026 15:07:40.453323 16390 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7088cfbe1e3e4e708cc7fd8afa3463ca" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 36573 } }
I20251026 15:07:40.453428 16390 leader_election.cc:304] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7088cfbe1e3e4e708cc7fd8afa3463ca; no voters:
I20251026 15:07:40.453573 16390 leader_election.cc:290] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:07:40.453603 16395 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:40.453699 16395 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [term 1 LEADER]: Becoming Leader. State: Replica: 7088cfbe1e3e4e708cc7fd8afa3463ca, State: Running, Role: LEADER
I20251026 15:07:40.453775 16395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7088cfbe1e3e4e708cc7fd8afa3463ca" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 36573 } }
I20251026 15:07:40.453801 16390 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:07:40.454070 16395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7088cfbe1e3e4e708cc7fd8afa3463ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7088cfbe1e3e4e708cc7fd8afa3463ca" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 36573 } } }
I20251026 15:07:40.454134 16395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [sys.catalog]: This master's current role is: LEADER
I20251026 15:07:40.454236 16395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7088cfbe1e3e4e708cc7fd8afa3463ca. Latest consensus state: current_term: 1 leader_uuid: "7088cfbe1e3e4e708cc7fd8afa3463ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7088cfbe1e3e4e708cc7fd8afa3463ca" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 36573 } } }
I20251026 15:07:40.454298 16395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca [sys.catalog]: This master's current role is: LEADER
I20251026 15:07:40.454478 16399 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:07:40.455015 16399 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:07:40.458190 16399 catalog_manager.cc:1357] Generated new cluster ID: 6cbe3c2ea084422c86f9e4853a0fef26
I20251026 15:07:40.458245 16399 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 26 15:07:40 dist-test-slave-8pgz krb5kdc[16292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:40.462709 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:40.453126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.14.212.126:36573)
1026 15:07:40.453270 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:40.453282 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:40.453301 (+ 19us) server_negotiation.cc:408] Connection header received
1026 15:07:40.453389 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:40.453396 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:40.453447 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:40.453576 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:40.454601 (+ 1025us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.455854 (+ 1253us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:40.456821 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.457047 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:40.459088 (+ 2041us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:40.459093 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:40.459094 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:40.459115 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:40.461206 (+ 2091us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:40.461568 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:40.461571 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:40.461571 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:40.461630 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:40.461789 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:40.461790 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:40.461790 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:40.462035 (+ 245us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:40.462180 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:40.462257 (+ 77us) server_negotiation.cc:300] Negotiation successful
1026 15:07:40.462512 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20251026 15:07:40.465270 16399 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:07:40.465770 16399 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:07:40.471773 16399 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca: Generated new TSK 0
I20251026 15:07:40.471926 16399 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:40.522986 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:36573
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:40.617385 16420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:40.617573 16420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:40.617596 16420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:40.619046 16420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:40.619091 16420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:40.619112 16420 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 15:07:40.619130 16420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:07:40.620626 16420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.212.126:36573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:40.620862 16420 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:40.621062 16420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:07:40.623543 16420 server_base.cc:1047] running on GCE node
W20251026 15:07:40.623509 16425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:40.623683 16428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:40.623507 16426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:40.623929 16420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:40.624150 16420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:40.625324 16420 hybrid_clock.cc:648] HybridClock initialized: now 1761491260625270 us; error 75 us; skew 500 ppm
Oct 26 15:07:40 dist-test-slave-8pgz krb5kdc[16292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491260, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:40.628216 16420 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:07:40.628837 16420 webserver.cc:492] Webserver started at http://127.14.212.65:34111/ using document root <none> and password file <none>
I20251026 15:07:40.629077 16420 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:40.629122 16420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:40.629230 16420 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:40.630476 16420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "3760ef5ebc15404a933e6b314afea2d0"
format_stamp: "Formatted at 2025-10-26 15:07:40 on dist-test-slave-8pgz"
server_key: "2ef41300e558aadb6fdc57aaa6869023"
server_key_iv: "0133c6790952c8c896ef1917222f2607"
server_key_version: "encryptionkey@0"
I20251026 15:07:40.630831 16420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "3760ef5ebc15404a933e6b314afea2d0"
format_stamp: "Formatted at 2025-10-26 15:07:40 on dist-test-slave-8pgz"
server_key: "2ef41300e558aadb6fdc57aaa6869023"
server_key_iv: "0133c6790952c8c896ef1917222f2607"
server_key_version: "encryptionkey@0"
I20251026 15:07:40.632315 16420 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:07:40.633121 16435 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:40.633301 16420 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:07:40.633380 16420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "3760ef5ebc15404a933e6b314afea2d0"
format_stamp: "Formatted at 2025-10-26 15:07:40 on dist-test-slave-8pgz"
server_key: "2ef41300e558aadb6fdc57aaa6869023"
server_key_iv: "0133c6790952c8c896ef1917222f2607"
server_key_version: "encryptionkey@0"
I20251026 15:07:40.633442 16420 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:40.650857 16420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:40.653182 16420 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:40.653337 16420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:40.653811 16420 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:40.653851 16420 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:40.653873 16420 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:40.653887 16420 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 15:07:40 dist-test-slave-8pgz krb5kdc[16292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491260, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:40.660887 16420 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:38713
I20251026 15:07:40.661015 16554 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:38713 every 8 connection(s)
I20251026 15:07:40.661252 16420 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:07:40.663815 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:40.654237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:37369 (local address 127.14.212.126:36573)
1026 15:07:40.654306 (+ 69us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:40.654309 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:40.654463 (+ 154us) server_negotiation.cc:408] Connection header received
1026 15:07:40.655025 (+ 562us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:40.655026 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:40.655040 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:40.655120 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:40.656168 (+ 1048us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.657020 (+ 852us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:40.657637 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.657847 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:40.661334 (+ 3487us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:40.661340 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:40.661340 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:40.661349 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:40.662791 (+ 1442us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:40.663151 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:40.663152 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:40.663152 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:40.663173 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:40.663318 (+ 145us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:40.663319 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:40.663319 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:40.663411 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:40.663461 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:40.663668 (+ 207us) server_negotiation.cc:300] Negotiation successful
1026 15:07:40.663748 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251026 15:07:40.663925 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:40.654171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:36573 (local address 127.14.212.65:37369)
1026 15:07:40.654341 (+ 170us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:40.654370 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:40.654921 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:40.655170 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:40.655176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:40.655612 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:40.656051 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:40.656057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.657117 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:40.657118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:40.657555 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:40.657557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.657704 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:40.658506 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:40.658534 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:40.661234 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:40.662880 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:40.662886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:40.662888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:40.663084 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:40.663221 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:40.663222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:40.663222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:40.663275 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:40.663455 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:40.663458 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:40.663570 (+ 112us) client_negotiation.cc:769] Sending connection context
1026 15:07:40.663653 (+ 83us) client_negotiation.cc:240] Negotiation successful
1026 15:07:40.663770 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20251026 15:07:40.664449 16555 heartbeater.cc:344] Connected to a master server at 127.14.212.126:36573
I20251026 15:07:40.664530 16555 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:40.664780 16555 heartbeater.cc:507] Master 127.14.212.126:36573 requested a full tablet report, sending...
I20251026 15:07:40.665256 16342 ts_manager.cc:194] Registered new tserver with Master: 3760ef5ebc15404a933e6b314afea2d0 (127.14.212.65:38713)
I20251026 15:07:40.666416 16342 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:37369
I20251026 15:07:40.668011 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 16420
I20251026 15:07:40.668097 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:07:40.668299 15185 external_mini_cluster.cc:1468] Setting key 04de392acf7280f145f67d808cacba09
I20251026 15:07:40.671152 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:40.665502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:53557 (local address 127.14.212.126:36573)
1026 15:07:40.665539 (+ 37us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:40.665541 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:40.665643 (+ 102us) server_negotiation.cc:408] Connection header received
1026 15:07:40.665703 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:40.665704 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:40.665712 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:40.665776 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:40.666290 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.666867 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:40.667392 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.667516 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:40.668289 (+ 773us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:40.668292 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:40.668292 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:40.668298 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:40.669867 (+ 1569us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:40.670263 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:40.670264 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:40.670264 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:40.670287 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:40.670677 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:40.670678 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:40.670678 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:40.670773 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:40.670838 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:40.670976 (+ 138us) server_negotiation.cc:300] Negotiation successful
1026 15:07:40.671063 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20251026 15:07:40.671167 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:40.665448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:36573 (local address 127.14.212.65:53557)
1026 15:07:40.665578 (+ 130us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:40.665589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:40.665648 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:40.665799 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:40.665801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:40.665967 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:40.666217 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:40.666219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.666941 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:40.666942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:40.667306 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:40.667308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.667367 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:40.667828 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:40.667833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:40.668211 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:40.669968 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:40.669969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:40.669970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:40.670172 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:40.670363 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:40.670364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:40.670364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:40.670585 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:40.670835 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:40.670836 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:40.670881 (+ 45us) client_negotiation.cc:769] Sending connection context
1026 15:07:40.670950 (+ 69us) client_negotiation.cc:240] Negotiation successful
1026 15:07:40.671063 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:40.711544 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:36573
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:40.789776 16566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:40.789954 16566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:40.789978 16566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:40.791342 16566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:40.791384 16566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:40.791404 16566 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 15:07:40.791421 16566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:07:40.792953 16566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.212.126:36573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:40.793144 16566 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:40.793294 16566 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:40.795828 16574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:40.795833 16571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:40.795867 16572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:40.796051 16566 server_base.cc:1047] running on GCE node
I20251026 15:07:40.796205 16566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:40.796396 16566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:40.797534 16566 hybrid_clock.cc:648] HybridClock initialized: now 1761491260797515 us; error 33 us; skew 500 ppm
Oct 26 15:07:40 dist-test-slave-8pgz krb5kdc[16292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491260, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:40.800027 16566 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:07:40.800588 16566 webserver.cc:492] Webserver started at http://127.14.212.66:35737/ using document root <none> and password file <none>
I20251026 15:07:40.800788 16566 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:40.800828 16566 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:40.800941 16566 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:40.801893 16566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "74e705caaa894ffc9f72333bc47e4768"
format_stamp: "Formatted at 2025-10-26 15:07:40 on dist-test-slave-8pgz"
server_key: "b1fe3a398621470ce396748239ec4d53"
server_key_iv: "308698415498e4961596325065876035"
server_key_version: "encryptionkey@0"
I20251026 15:07:40.802191 16566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "74e705caaa894ffc9f72333bc47e4768"
format_stamp: "Formatted at 2025-10-26 15:07:40 on dist-test-slave-8pgz"
server_key: "b1fe3a398621470ce396748239ec4d53"
server_key_iv: "308698415498e4961596325065876035"
server_key_version: "encryptionkey@0"
I20251026 15:07:40.803436 16566 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:07:40.804121 16581 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:40.804309 16566 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:07:40.804385 16566 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "74e705caaa894ffc9f72333bc47e4768"
format_stamp: "Formatted at 2025-10-26 15:07:40 on dist-test-slave-8pgz"
server_key: "b1fe3a398621470ce396748239ec4d53"
server_key_iv: "308698415498e4961596325065876035"
server_key_version: "encryptionkey@0"
I20251026 15:07:40.804447 16566 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:40.814496 16566 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:40.816586 16566 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:40.816735 16566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:40.817312 16566 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:40.817364 16566 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:40.817399 16566 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:40.817427 16566 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 15:07:40 dist-test-slave-8pgz krb5kdc[16292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491260, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:40.824625 16566 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:46713
I20251026 15:07:40.824705 16697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:46713 every 8 connection(s)
I20251026 15:07:40.825011 16566 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:07:40.825569 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 16566
I20251026 15:07:40.825673 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:07:40.825865 15185 external_mini_cluster.cc:1468] Setting key 9bd41013ac0b6d26c9bc5ea813c66779
I20251026 15:07:40.827248 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:40.817620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:36573 (local address 127.14.212.66:50683)
1026 15:07:40.817771 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:40.817802 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:40.818690 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:40.819047 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:40.819053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:40.819355 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:40.819642 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:40.819646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.820767 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:40.820769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:40.821174 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:40.821175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.821319 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:40.821904 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:40.821934 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:40.823908 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:40.825988 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:40.825995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:40.825998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:40.826230 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:40.826386 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:40.826387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:40.826388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:40.826479 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:40.826747 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:40.826752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:40.826880 (+ 128us) client_negotiation.cc:769] Sending connection context
1026 15:07:40.826989 (+ 109us) client_negotiation.cc:240] Negotiation successful
1026 15:07:40.827107 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":91,"threads_started":1}
I20251026 15:07:40.827289 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:40.817668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:50683 (local address 127.14.212.126:36573)
1026 15:07:40.817752 (+ 84us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:40.817756 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:40.817893 (+ 137us) server_negotiation.cc:408] Connection header received
1026 15:07:40.818848 (+ 955us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:40.818849 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:40.818866 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:40.818934 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:40.819751 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.820601 (+ 850us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:40.821260 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.821441 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:40.824040 (+ 2599us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:40.824046 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:40.824047 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:40.824055 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:40.825878 (+ 1823us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:40.826295 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:40.826296 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:40.826296 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:40.826324 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:40.826536 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:40.826537 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:40.826537 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:40.826657 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:40.826732 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:40.827043 (+ 311us) server_negotiation.cc:300] Negotiation successful
1026 15:07:40.827154 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251026 15:07:40.827780 16698 heartbeater.cc:344] Connected to a master server at 127.14.212.126:36573
I20251026 15:07:40.827872 16698 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:40.828130 16698 heartbeater.cc:507] Master 127.14.212.126:36573 requested a full tablet report, sending...
I20251026 15:07:40.828464 16342 ts_manager.cc:194] Registered new tserver with Master: 74e705caaa894ffc9f72333bc47e4768 (127.14.212.66:46713)
I20251026 15:07:40.828886 16342 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:50683
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
I20251026 15:07:40.835253 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:40.829048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:40351 (local address 127.14.212.126:36573)
1026 15:07:40.829090 (+ 42us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:40.829094 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:40.829193 (+ 99us) server_negotiation.cc:408] Connection header received
1026 15:07:40.829850 (+ 657us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:40.829851 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:40.829860 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:40.829931 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:40.830506 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.830993 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:40.831838 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.831968 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:40.832868 (+ 900us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:40.832872 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:40.832872 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:40.832879 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:40.834251 (+ 1372us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:40.834670 (+ 419us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:40.834672 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:40.834672 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:40.834703 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:40.834856 (+ 153us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:40.834856 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:40.834856 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:40.834969 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:40.835019 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:40.835122 (+ 103us) server_negotiation.cc:300] Negotiation successful
1026 15:07:40.835190 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251026 15:07:40.835265 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:40.828986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:36573 (local address 127.14.212.66:40351)
1026 15:07:40.829130 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:40.829141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:40.829762 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:40.829930 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:40.829932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:40.830129 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:40.830421 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:40.830424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.831090 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:40.831092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:40.831736 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:40.831739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:40.831825 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:40.832396 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:40.832401 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:40.832780 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:40.834340 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:40.834341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:40.834342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:40.834569 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:40.834776 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:40.834776 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:40.834777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:40.834808 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:40.835016 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:40.835016 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:40.835049 (+ 33us) client_negotiation.cc:769] Sending connection context
1026 15:07:40.835098 (+ 49us) client_negotiation.cc:240] Negotiation successful
1026 15:07:40.835181 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:40.877758 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:36573
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:40.957826 16709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:40.958011 16709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:40.958035 16709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:40.959458 16709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:40.959512 16709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:40.959534 16709 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 15:07:40.959553 16709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:07:40.961396 16709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.212.126:36573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:40.961658 16709 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:40.961864 16709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:40.964534 16714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:40.964524 16717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:40.964574 16709 server_base.cc:1047] running on GCE node
W20251026 15:07:40.964524 16715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:40.964864 16709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:40.965077 16709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:40.966219 16709 hybrid_clock.cc:648] HybridClock initialized: now 1761491260966190 us; error 48 us; skew 500 ppm
Oct 26 15:07:40 dist-test-slave-8pgz krb5kdc[16292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491260, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:40.968703 16709 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:07:40.969233 16709 webserver.cc:492] Webserver started at http://127.14.212.67:34163/ using document root <none> and password file <none>
I20251026 15:07:40.969424 16709 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:40.969465 16709 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:40.969576 16709 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:40.970588 16709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "8bc4759ae84349a696d99e7a75a75c63"
format_stamp: "Formatted at 2025-10-26 15:07:40 on dist-test-slave-8pgz"
server_key: "8a515cab8a03ffad0dec1d9b4f591265"
server_key_iv: "55e02f466b17302c3eca6ba0e4ed8b5c"
server_key_version: "encryptionkey@0"
I20251026 15:07:40.970913 16709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "8bc4759ae84349a696d99e7a75a75c63"
format_stamp: "Formatted at 2025-10-26 15:07:40 on dist-test-slave-8pgz"
server_key: "8a515cab8a03ffad0dec1d9b4f591265"
server_key_iv: "55e02f466b17302c3eca6ba0e4ed8b5c"
server_key_version: "encryptionkey@0"
I20251026 15:07:40.972148 16709 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:40.972893 16724 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:40.973088 16709 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:40.973165 16709 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "8bc4759ae84349a696d99e7a75a75c63"
format_stamp: "Formatted at 2025-10-26 15:07:40 on dist-test-slave-8pgz"
server_key: "8a515cab8a03ffad0dec1d9b4f591265"
server_key_iv: "55e02f466b17302c3eca6ba0e4ed8b5c"
server_key_version: "encryptionkey@0"
I20251026 15:07:40.973225 16709 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:41.010031 16709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:41.012090 16709 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:41.012238 16709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:41.012820 16709 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:41.012861 16709 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:41.012917 16709 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:41.012949 16709 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 15:07:41 dist-test-slave-8pgz krb5kdc[16292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491260, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:41.019923 16709 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:38325
I20251026 15:07:41.019970 16840 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:38325 every 8 connection(s)
I20251026 15:07:41.020319 16709 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:07:41.022622 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.013241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:54599 (local address 127.14.212.126:36573)
1026 15:07:41.013362 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.013366 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.013638 (+ 272us) server_negotiation.cc:408] Connection header received
1026 15:07:41.014247 (+ 609us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.014249 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.014266 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.014345 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:41.015408 (+ 1063us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.016271 (+ 863us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.017114 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.017339 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.019864 (+ 2525us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:41.019869 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:41.019870 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:41.019878 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:41.021449 (+ 1571us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:41.021837 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:41.021838 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:41.021838 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:41.021862 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:41.022040 (+ 178us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:41.022041 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:41.022041 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:41.022135 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:41.022195 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.022452 (+ 257us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.022534 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20251026 15:07:41.022789 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.013219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:36573 (local address 127.14.212.67:54599)
1026 15:07:41.013436 (+ 217us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:41.013527 (+ 91us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:41.014106 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:41.014440 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:41.014445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:41.014870 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:41.015298 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.015304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.016367 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.016371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:41.017022 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.017025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.017185 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.017888 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:41.017912 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:41.019760 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:41.021534 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:41.021540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:41.021543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:41.021765 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:41.021917 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:41.021918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:41.021918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:41.021982 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:41.022190 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:41.022195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:41.022316 (+ 121us) client_negotiation.cc:769] Sending connection context
1026 15:07:41.022436 (+ 120us) client_negotiation.cc:240] Negotiation successful
1026 15:07:41.022586 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20251026 15:07:41.022886 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 16709
I20251026 15:07:41.022972 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:07:41.023118 15185 external_mini_cluster.cc:1468] Setting key a07b7681a029d58727c637b16573384f
I20251026 15:07:41.023507 16841 heartbeater.cc:344] Connected to a master server at 127.14.212.126:36573
I20251026 15:07:41.023597 16841 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:41.023866 16841 heartbeater.cc:507] Master 127.14.212.126:36573 requested a full tablet report, sending...
I20251026 15:07:41.024207 16342 ts_manager.cc:194] Registered new tserver with Master: 8bc4759ae84349a696d99e7a75a75c63 (127.14.212.67:38325)
I20251026 15:07:41.024675 16342 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:54599
I20251026 15:07:41.029196 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.024612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:50557 (local address 127.14.212.126:36573)
1026 15:07:41.024656 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.024658 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.024782 (+ 124us) server_negotiation.cc:408] Connection header received
1026 15:07:41.024858 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.024858 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.024868 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.024921 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:41.025321 (+ 400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.025775 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.026276 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.026483 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.027105 (+ 622us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:41.027108 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:41.027109 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:41.027114 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:41.028248 (+ 1134us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:41.028543 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:41.028544 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:41.028545 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:41.028568 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:41.028735 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:41.028736 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:41.028736 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:41.028829 (+ 93us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:41.028889 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.029031 (+ 142us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.029111 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251026 15:07:41.029213 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.024546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:36573 (local address 127.14.212.67:50557)
1026 15:07:41.024715 (+ 169us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:41.024724 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:41.024805 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:41.024929 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:41.024930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:41.025084 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:41.025248 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.025250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.025844 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.025845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:41.026206 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.026207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.026273 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.026769 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:41.026773 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:41.027027 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:41.028322 (+ 1295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:41.028323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:41.028324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:41.028479 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:41.028621 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:41.028622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:41.028622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:41.028649 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:41.028886 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:41.028887 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:41.028929 (+ 42us) client_negotiation.cc:769] Sending connection context
1026 15:07:41.029002 (+ 73us) client_negotiation.cc:240] Negotiation successful
1026 15:07:41.029103 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20251026 15:07:41.034961 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 26 15:07:41 dist-test-slave-8pgz krb5kdc[16292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491261, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 26 15:07:41 dist-test-slave-8pgz krb5kdc[16292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491261, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:41.055475 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.049052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56332 (local address 127.14.212.126:36573)
1026 15:07:41.049112 (+ 60us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.049114 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.049144 (+ 30us) server_negotiation.cc:408] Connection header received
1026 15:07:41.049168 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.049169 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.049178 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.049219 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:41.049857 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.050343 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.050930 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.051080 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.053099 (+ 2019us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:41.053104 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:41.053104 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:41.053111 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:41.054637 (+ 1526us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:41.054948 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:41.054948 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:41.054949 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:41.054966 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:41.055066 (+ 100us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:41.055066 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:41.055067 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:41.055147 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:41.055217 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.055310 (+ 93us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.055373 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251026 15:07:41.056946 16342 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:07:41.058183 16342 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:07:41.063717 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.060427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46713 (local address 127.0.0.1:49920)
1026 15:07:41.061052 (+ 625us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:41.061061 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:41.061112 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:41.061298 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:41.061299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:41.061300 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:41.061478 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.061480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.062793 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.062794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:41.063465 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.063470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.063533 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.063576 (+ 43us) client_negotiation.cc:769] Sending connection context
1026 15:07:41.063627 (+ 51us) client_negotiation.cc:240] Negotiation successful
1026 15:07:41.063660 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":368,"thread_start_us":70,"threads_started":1}
I20251026 15:07:41.063815 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.060428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:38713 (local address 127.0.0.1:42058)
1026 15:07:41.060689 (+ 261us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:41.060718 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:41.060811 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:41.061052 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:41.061054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:41.061057 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:41.061328 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.061331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.062751 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.062751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:41.063572 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.063574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.063649 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.063652 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:07:41.063695 (+ 43us) client_negotiation.cc:240] Negotiation successful
1026 15:07:41.063740 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":60,"threads_started":1}
I20251026 15:07:41.064281 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.060429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:38325 (local address 127.0.0.1:42704)
1026 15:07:41.060918 (+ 489us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:41.060928 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:41.060988 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:41.061216 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:41.061217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:41.061219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:41.061598 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.061601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.062630 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.062632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:41.063705 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.063707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.063766 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.064133 (+ 367us) client_negotiation.cc:769] Sending connection context
1026 15:07:41.064177 (+ 44us) client_negotiation.cc:240] Negotiation successful
1026 15:07:41.064219 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":4,"thread_start_us":98,"threads_started":1}
I20251026 15:07:41.064517 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.060665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.14.212.67:38325)
1026 15:07:41.060880 (+ 215us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.060888 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.060994 (+ 106us) server_negotiation.cc:408] Connection header received
1026 15:07:41.061053 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.061056 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.061070 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.061128 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:41.061657 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.062546 (+ 889us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.063802 (+ 1256us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.064308 (+ 506us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.064369 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.064401 (+ 32us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.064442 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":96,"threads_started":1}
I20251026 15:07:41.064582 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.060546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.14.212.66:46713)
1026 15:07:41.060944 (+ 398us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.060971 (+ 27us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.061165 (+ 194us) server_negotiation.cc:408] Connection header received
1026 15:07:41.061204 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.061206 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.061220 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.061446 (+ 226us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:41.061547 (+ 101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.062691 (+ 1144us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.063935 (+ 1244us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.064398 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.064437 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.064464 (+ 27us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.064501 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":68,"threads_started":1}
I20251026 15:07:41.064777 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.060518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.14.212.65:38713)
1026 15:07:41.060671 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.060679 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.060833 (+ 154us) server_negotiation.cc:408] Connection header received
1026 15:07:41.060916 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.060918 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.060955 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.061034 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:41.061415 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.062607 (+ 1192us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.064000 (+ 1393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.064541 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.064608 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.064642 (+ 34us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.064700 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20251026 15:07:41.065042 16775 tablet_service.cc:1505] Processing CreateTablet for tablet 9d4549b470994b2ea69216848e13395f (DEFAULT_TABLE table=test-table [id=def01494595a4a6e8f72c1aca83b9edc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:41.065326 16775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d4549b470994b2ea69216848e13395f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:41.065263 16487 tablet_service.cc:1505] Processing CreateTablet for tablet 9d4549b470994b2ea69216848e13395f (DEFAULT_TABLE table=test-table [id=def01494595a4a6e8f72c1aca83b9edc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:41.065526 16487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d4549b470994b2ea69216848e13395f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:41.066855 16632 tablet_service.cc:1505] Processing CreateTablet for tablet 9d4549b470994b2ea69216848e13395f (DEFAULT_TABLE table=test-table [id=def01494595a4a6e8f72c1aca83b9edc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:41.067129 16632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d4549b470994b2ea69216848e13395f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:41.067660 16864 tablet_bootstrap.cc:492] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63: Bootstrap starting.
I20251026 15:07:41.068084 16865 tablet_bootstrap.cc:492] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0: Bootstrap starting.
I20251026 15:07:41.068303 16864 tablet_bootstrap.cc:654] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:41.068665 16865 tablet_bootstrap.cc:654] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:41.069157 16864 log.cc:826] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:41.069327 16865 log.cc:826] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:41.069865 16866 tablet_bootstrap.cc:492] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768: Bootstrap starting.
I20251026 15:07:41.070050 16865 tablet_bootstrap.cc:492] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0: No bootstrap required, opened a new log
I20251026 15:07:41.070109 16865 ts_tablet_manager.cc:1403] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:07:41.070382 16866 tablet_bootstrap.cc:654] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:41.070886 16864 tablet_bootstrap.cc:492] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63: No bootstrap required, opened a new log
I20251026 15:07:41.070961 16864 ts_tablet_manager.cc:1403] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 15:07:41.070994 16866 log.cc:826] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:41.071555 16865 raft_consensus.cc:359] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } }
I20251026 15:07:41.071674 16865 raft_consensus.cc:385] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:41.071714 16865 raft_consensus.cc:740] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3760ef5ebc15404a933e6b314afea2d0, State: Initialized, Role: FOLLOWER
I20251026 15:07:41.071738 16866 tablet_bootstrap.cc:492] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768: No bootstrap required, opened a new log
I20251026 15:07:41.071790 16866 ts_tablet_manager.cc:1403] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:07:41.071805 16865 consensus_queue.cc:260] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } }
I20251026 15:07:41.072034 16865 ts_tablet_manager.cc:1434] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:07:41.072171 16555 heartbeater.cc:499] Master 127.14.212.126:36573 was elected leader, sending a full tablet report...
I20251026 15:07:41.072510 16864 raft_consensus.cc:359] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } }
I20251026 15:07:41.072659 16864 raft_consensus.cc:385] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:41.072702 16864 raft_consensus.cc:740] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc4759ae84349a696d99e7a75a75c63, State: Initialized, Role: FOLLOWER
I20251026 15:07:41.072785 16864 consensus_queue.cc:260] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } }
I20251026 15:07:41.073021 16864 ts_tablet_manager.cc:1434] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:07:41.073122 16841 heartbeater.cc:499] Master 127.14.212.126:36573 was elected leader, sending a full tablet report...
I20251026 15:07:41.073431 16866 raft_consensus.cc:359] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } }
I20251026 15:07:41.073532 16866 raft_consensus.cc:385] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:41.073558 16866 raft_consensus.cc:740] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e705caaa894ffc9f72333bc47e4768, State: Initialized, Role: FOLLOWER
I20251026 15:07:41.073652 16866 consensus_queue.cc:260] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } }
I20251026 15:07:41.073876 16866 ts_tablet_manager.cc:1434] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:07:41.073997 16698 heartbeater.cc:499] Master 127.14.212.126:36573 was elected leader, sending a full tablet report...
W20251026 15:07:41.075469 16699 tablet.cc:2378] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 15:07:41.162324 16556 tablet.cc:2378] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:07:41.222491 16871 raft_consensus.cc:493] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:07:41.222626 16871 raft_consensus.cc:515] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } }
I20251026 15:07:41.223006 16871 leader_election.cc:290] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3760ef5ebc15404a933e6b314afea2d0 (127.14.212.65:38713), 74e705caaa894ffc9f72333bc47e4768 (127.14.212.66:46713)
I20251026 15:07:41.225927 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.223209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46713 (local address 127.14.212.67:47603)
1026 15:07:41.223277 (+ 68us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:41.223290 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:41.223361 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:41.223515 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:41.223516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:41.223518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:41.223753 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.223755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.224697 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.224698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:41.225668 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.225670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.225753 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.225759 (+ 6us) client_negotiation.cc:769] Sending connection context
1026 15:07:41.225795 (+ 36us) client_negotiation.cc:240] Negotiation successful
1026 15:07:41.225842 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20251026 15:07:41.225934 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.223208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:38713 (local address 127.14.212.67:50955)
1026 15:07:41.223448 (+ 240us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:41.223456 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:41.223515 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:41.223630 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:41.223631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:41.223632 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:41.223886 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.223887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.224759 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.224760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:41.225736 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.225738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.225800 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.225802 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:07:41.225834 (+ 32us) client_negotiation.cc:240] Negotiation successful
1026 15:07:41.225859 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":58,"thread_start_us":109,"threads_started":1}
I20251026 15:07:41.226372 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.223321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:47603 (local address 127.14.212.66:46713)
1026 15:07:41.223372 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.223374 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.223388 (+ 14us) server_negotiation.cc:408] Connection header received
1026 15:07:41.223422 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.223423 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.223434 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.223497 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:41.223836 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.224618 (+ 782us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.225766 (+ 1148us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.226243 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.226252 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.226274 (+ 22us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.226306 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251026 15:07:41.226372 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.223290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:50955 (local address 127.14.212.65:38713)
1026 15:07:41.223336 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.223338 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.223527 (+ 189us) server_negotiation.cc:408] Connection header received
1026 15:07:41.223562 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.223563 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.223570 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.223615 (+ 45us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:41.223965 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.224694 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.225813 (+ 1119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.226248 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.226253 (+ 5us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.226274 (+ 21us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.226306 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251026 15:07:41.226691 16507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d4549b470994b2ea69216848e13395f" candidate_uuid: "8bc4759ae84349a696d99e7a75a75c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3760ef5ebc15404a933e6b314afea2d0" is_pre_election: true
I20251026 15:07:41.226728 16652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d4549b470994b2ea69216848e13395f" candidate_uuid: "8bc4759ae84349a696d99e7a75a75c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e705caaa894ffc9f72333bc47e4768" is_pre_election: true
I20251026 15:07:41.226851 16507 raft_consensus.cc:2468] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bc4759ae84349a696d99e7a75a75c63 in term 0.
I20251026 15:07:41.226850 16652 raft_consensus.cc:2468] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bc4759ae84349a696d99e7a75a75c63 in term 0.
I20251026 15:07:41.227080 16726 leader_election.cc:304] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74e705caaa894ffc9f72333bc47e4768, 8bc4759ae84349a696d99e7a75a75c63; no voters:
I20251026 15:07:41.227252 16871 raft_consensus.cc:2804] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:07:41.227288 16871 raft_consensus.cc:493] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:07:41.227304 16871 raft_consensus.cc:3060] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:41.227967 16871 raft_consensus.cc:515] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } }
I20251026 15:07:41.228091 16871 leader_election.cc:290] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [CANDIDATE]: Term 1 election: Requested vote from peers 3760ef5ebc15404a933e6b314afea2d0 (127.14.212.65:38713), 74e705caaa894ffc9f72333bc47e4768 (127.14.212.66:46713)
I20251026 15:07:41.228250 16652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d4549b470994b2ea69216848e13395f" candidate_uuid: "8bc4759ae84349a696d99e7a75a75c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e705caaa894ffc9f72333bc47e4768"
I20251026 15:07:41.228277 16507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d4549b470994b2ea69216848e13395f" candidate_uuid: "8bc4759ae84349a696d99e7a75a75c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3760ef5ebc15404a933e6b314afea2d0"
I20251026 15:07:41.228320 16652 raft_consensus.cc:3060] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:41.228358 16507 raft_consensus.cc:3060] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:41.229092 16507 raft_consensus.cc:2468] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bc4759ae84349a696d99e7a75a75c63 in term 1.
I20251026 15:07:41.229120 16652 raft_consensus.cc:2468] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bc4759ae84349a696d99e7a75a75c63 in term 1.
I20251026 15:07:41.229281 16726 leader_election.cc:304] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74e705caaa894ffc9f72333bc47e4768, 8bc4759ae84349a696d99e7a75a75c63; no voters:
I20251026 15:07:41.229367 16871 raft_consensus.cc:2804] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:41.229530 16871 raft_consensus.cc:697] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [term 1 LEADER]: Becoming Leader. State: Replica: 8bc4759ae84349a696d99e7a75a75c63, State: Running, Role: LEADER
I20251026 15:07:41.229616 16871 consensus_queue.cc:237] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } }
I20251026 15:07:41.230427 16342 catalog_manager.cc:5649] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bc4759ae84349a696d99e7a75a75c63 (127.14.212.67). New cstate: current_term: 1 leader_uuid: "8bc4759ae84349a696d99e7a75a75c63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } health_report { overall_health: UNKNOWN } } }
I20251026 15:07:41.248049 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.245752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.14.212.67:38325)
1026 15:07:41.245825 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.245827 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.245836 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:07:41.245919 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.245920 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.245937 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.245999 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:41.246331 (+ 332us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.246900 (+ 569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.247482 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.247594 (+ 112us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.247626 (+ 32us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:41.247720 (+ 94us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:41.247793 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.247923 (+ 130us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.247952 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251026 15:07:41.249948 16652 raft_consensus.cc:1275] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768 [term 1 FOLLOWER]: Refusing update from remote peer 8bc4759ae84349a696d99e7a75a75c63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:41.249944 16507 raft_consensus.cc:1275] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0 [term 1 FOLLOWER]: Refusing update from remote peer 8bc4759ae84349a696d99e7a75a75c63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:41.250290 16874 consensus_queue.cc:1048] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:41.250413 16871 consensus_queue.cc:1048] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:41.253727 16880 mvcc.cc:204] Tried to move back new op lower bound from 7215068206076928000 to 7215068205997047808. Current Snapshot: MvccSnapshot[applied={T|T < 7215068206076928000}]
I20251026 15:07:41.256256 16342 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56332:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:07:41.256367 16342 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56332:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:07:41.257184 16342 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca: Sending DeleteTablet for 3 replicas of tablet 9d4549b470994b2ea69216848e13395f
I20251026 15:07:41.257550 16632 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d4549b470994b2ea69216848e13395f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:49920
I20251026 15:07:41.257555 16775 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d4549b470994b2ea69216848e13395f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:42704
I20251026 15:07:41.257608 16487 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d4549b470994b2ea69216848e13395f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:42058
I20251026 15:07:41.257834 16888 tablet_replica.cc:333] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768: stopping tablet replica
I20251026 15:07:41.257802 16342 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 15:07:41.257928 16888 raft_consensus.cc:2243] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:07:41.257942 16889 tablet_replica.cc:333] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63: stopping tablet replica
W20251026 15:07:41.257943 16342 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:07:41.258016 16889 raft_consensus.cc:2243] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:07:41.258030 16888 raft_consensus.cc:2272] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:41.258581 16888 ts_tablet_manager.cc:1916] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:41.259471 16487 tablet_service.cc:1505] Processing CreateTablet for tablet 1c9e804c9a63493eb3f71807e52f037c (DEFAULT_TABLE table=test-table [id=97524128c36b424cada08934f51e96c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:41.259496 16775 tablet_service.cc:1505] Processing CreateTablet for tablet 1c9e804c9a63493eb3f71807e52f037c (DEFAULT_TABLE table=test-table [id=97524128c36b424cada08934f51e96c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:41.259595 16487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c9e804c9a63493eb3f71807e52f037c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:41.259601 16775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c9e804c9a63493eb3f71807e52f037c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:41.260423 16888 ts_tablet_manager.cc:1929] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:07:41.260483 16888 log.cc:1199] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/9d4549b470994b2ea69216848e13395f
I20251026 15:07:41.260740 16888 ts_tablet_manager.cc:1950] T 9d4549b470994b2ea69216848e13395f P 74e705caaa894ffc9f72333bc47e4768: Deleting consensus metadata
I20251026 15:07:41.260968 16865 tablet_bootstrap.cc:492] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0: Bootstrap starting.
I20251026 15:07:41.261031 16864 tablet_bootstrap.cc:492] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63: Bootstrap starting.
I20251026 15:07:41.261413 16864 tablet_bootstrap.cc:654] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:41.261413 16865 tablet_bootstrap.cc:654] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:41.262019 16327 catalog_manager.cc:4985] TS 74e705caaa894ffc9f72333bc47e4768 (127.14.212.66:46713): tablet 9d4549b470994b2ea69216848e13395f (table test-table [id=def01494595a4a6e8f72c1aca83b9edc]) successfully deleted
I20251026 15:07:41.262084 16632 tablet_service.cc:1505] Processing CreateTablet for tablet 1c9e804c9a63493eb3f71807e52f037c (DEFAULT_TABLE table=test-table [id=97524128c36b424cada08934f51e96c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:41.262200 16632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c9e804c9a63493eb3f71807e52f037c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:41.262475 16889 raft_consensus.cc:2272] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:41.262934 16889 ts_tablet_manager.cc:1916] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:41.263540 16866 tablet_bootstrap.cc:492] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768: Bootstrap starting.
I20251026 15:07:41.263805 16865 tablet_bootstrap.cc:492] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0: No bootstrap required, opened a new log
I20251026 15:07:41.263855 16865 ts_tablet_manager.cc:1403] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251026 15:07:41.264004 16866 tablet_bootstrap.cc:654] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:41.264037 16865 raft_consensus.cc:359] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } }
I20251026 15:07:41.264106 16865 raft_consensus.cc:385] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:41.264124 16865 raft_consensus.cc:740] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3760ef5ebc15404a933e6b314afea2d0, State: Initialized, Role: FOLLOWER
I20251026 15:07:41.264168 16865 consensus_queue.cc:260] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } }
I20251026 15:07:41.264246 16865 ts_tablet_manager.cc:1434] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:41.264612 16889 ts_tablet_manager.cc:1929] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:07:41.264679 16889 log.cc:1199] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/9d4549b470994b2ea69216848e13395f
I20251026 15:07:41.265009 16889 ts_tablet_manager.cc:1950] T 9d4549b470994b2ea69216848e13395f P 8bc4759ae84349a696d99e7a75a75c63: Deleting consensus metadata
I20251026 15:07:41.265026 16864 tablet_bootstrap.cc:492] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63: No bootstrap required, opened a new log
I20251026 15:07:41.265082 16864 ts_tablet_manager.cc:1403] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20251026 15:07:41.265239 16864 raft_consensus.cc:359] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } }
I20251026 15:07:41.265307 16864 raft_consensus.cc:385] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:41.265326 16864 raft_consensus.cc:740] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc4759ae84349a696d99e7a75a75c63, State: Initialized, Role: FOLLOWER
I20251026 15:07:41.265370 16864 consensus_queue.cc:260] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } }
I20251026 15:07:41.265450 16866 tablet_bootstrap.cc:492] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768: No bootstrap required, opened a new log
I20251026 15:07:41.265471 16864 ts_tablet_manager.cc:1434] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:41.265494 16866 ts_tablet_manager.cc:1403] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:07:41.265637 16866 raft_consensus.cc:359] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } }
I20251026 15:07:41.265694 16866 raft_consensus.cc:385] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:41.265715 16866 raft_consensus.cc:740] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e705caaa894ffc9f72333bc47e4768, State: Initialized, Role: FOLLOWER
I20251026 15:07:41.265758 16866 consensus_queue.cc:260] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } }
I20251026 15:07:41.265841 16866 ts_tablet_manager.cc:1434] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:41.265887 16326 catalog_manager.cc:4985] TS 8bc4759ae84349a696d99e7a75a75c63 (127.14.212.67:38325): tablet 9d4549b470994b2ea69216848e13395f (table test-table [id=def01494595a4a6e8f72c1aca83b9edc]) successfully deleted
I20251026 15:07:41.266001 16887 tablet_replica.cc:333] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0: stopping tablet replica
I20251026 15:07:41.266103 16887 raft_consensus.cc:2243] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:07:41.266229 16887 raft_consensus.cc:2272] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:41.266667 16887 ts_tablet_manager.cc:1916] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:41.267988 16887 ts_tablet_manager.cc:1929] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:07:41.268047 16887 log.cc:1199] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/wal/wals/9d4549b470994b2ea69216848e13395f
I20251026 15:07:41.268244 16887 ts_tablet_manager.cc:1950] T 9d4549b470994b2ea69216848e13395f P 3760ef5ebc15404a933e6b314afea2d0: Deleting consensus metadata
I20251026 15:07:41.268594 16329 catalog_manager.cc:4985] TS 3760ef5ebc15404a933e6b314afea2d0 (127.14.212.65:38713): tablet 9d4549b470994b2ea69216848e13395f (table test-table [id=def01494595a4a6e8f72c1aca83b9edc]) successfully deleted
W20251026 15:07:41.270812 16842 tablet.cc:2378] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:07:41.686408 16874 raft_consensus.cc:493] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:07:41.686523 16874 raft_consensus.cc:515] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } }
I20251026 15:07:41.686688 16874 leader_election.cc:290] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3760ef5ebc15404a933e6b314afea2d0 (127.14.212.65:38713), 74e705caaa894ffc9f72333bc47e4768 (127.14.212.66:46713)
I20251026 15:07:41.687009 16652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c9e804c9a63493eb3f71807e52f037c" candidate_uuid: "8bc4759ae84349a696d99e7a75a75c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e705caaa894ffc9f72333bc47e4768" is_pre_election: true
I20251026 15:07:41.687018 16507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c9e804c9a63493eb3f71807e52f037c" candidate_uuid: "8bc4759ae84349a696d99e7a75a75c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3760ef5ebc15404a933e6b314afea2d0" is_pre_election: true
I20251026 15:07:41.687098 16652 raft_consensus.cc:2468] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bc4759ae84349a696d99e7a75a75c63 in term 0.
I20251026 15:07:41.687103 16507 raft_consensus.cc:2468] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bc4759ae84349a696d99e7a75a75c63 in term 0.
I20251026 15:07:41.687248 16726 leader_election.cc:304] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74e705caaa894ffc9f72333bc47e4768, 8bc4759ae84349a696d99e7a75a75c63; no voters:
I20251026 15:07:41.687354 16874 raft_consensus.cc:2804] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:07:41.687405 16874 raft_consensus.cc:493] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:07:41.687422 16874 raft_consensus.cc:3060] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:41.688171 16874 raft_consensus.cc:515] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } }
I20251026 15:07:41.688299 16874 leader_election.cc:290] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [CANDIDATE]: Term 1 election: Requested vote from peers 3760ef5ebc15404a933e6b314afea2d0 (127.14.212.65:38713), 74e705caaa894ffc9f72333bc47e4768 (127.14.212.66:46713)
I20251026 15:07:41.688459 16507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c9e804c9a63493eb3f71807e52f037c" candidate_uuid: "8bc4759ae84349a696d99e7a75a75c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3760ef5ebc15404a933e6b314afea2d0"
I20251026 15:07:41.688516 16507 raft_consensus.cc:3060] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:41.688467 16652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c9e804c9a63493eb3f71807e52f037c" candidate_uuid: "8bc4759ae84349a696d99e7a75a75c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e705caaa894ffc9f72333bc47e4768"
I20251026 15:07:41.688547 16652 raft_consensus.cc:3060] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:41.689114 16507 raft_consensus.cc:2468] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bc4759ae84349a696d99e7a75a75c63 in term 1.
I20251026 15:07:41.689175 16652 raft_consensus.cc:2468] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bc4759ae84349a696d99e7a75a75c63 in term 1.
I20251026 15:07:41.689302 16728 leader_election.cc:304] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3760ef5ebc15404a933e6b314afea2d0, 8bc4759ae84349a696d99e7a75a75c63; no voters:
I20251026 15:07:41.689407 16874 raft_consensus.cc:2804] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:41.689458 16874 raft_consensus.cc:697] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [term 1 LEADER]: Becoming Leader. State: Replica: 8bc4759ae84349a696d99e7a75a75c63, State: Running, Role: LEADER
I20251026 15:07:41.689496 16874 consensus_queue.cc:237] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } }
I20251026 15:07:41.690114 16342 catalog_manager.cc:5649] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bc4759ae84349a696d99e7a75a75c63 (127.14.212.67). New cstate: current_term: 1 leader_uuid: "8bc4759ae84349a696d99e7a75a75c63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } health_report { overall_health: HEALTHY } } }
Oct 26 15:07:41 dist-test-slave-8pgz krb5kdc[16292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491260, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:41.725780 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.719299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.14.212.126:36573)
1026 15:07:41.719430 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.719432 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.719445 (+ 13us) server_negotiation.cc:408] Connection header received
1026 15:07:41.719470 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.719471 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.719480 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.719538 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:41.720192 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.720705 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.721206 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.721324 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.723298 (+ 1974us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:41.723302 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:41.723302 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:41.723309 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:41.724855 (+ 1546us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:41.725190 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:41.725190 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:41.725190 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:41.725207 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:41.725372 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:41.725372 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:41.725373 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:41.725453 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:41.725498 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.725617 (+ 119us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.725687 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20251026 15:07:41.725893 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.719203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:36573 (local address 127.0.0.1:56336)
1026 15:07:41.719355 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:41.719364 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:41.719428 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:41.719552 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:41.719553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:41.719919 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:41.720087 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.720089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.720777 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.720778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:41.721137 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.721139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.721190 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.721784 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:41.721788 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:41.723210 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:41.724936 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:41.724943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:41.724945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:41.725127 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:41.725281 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:41.725282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:41.725282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:41.725314 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:41.725496 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:41.725502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:41.725543 (+ 41us) client_negotiation.cc:769] Sending connection context
1026 15:07:41.725599 (+ 56us) client_negotiation.cc:240] Negotiation successful
1026 15:07:41.725760 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":60,"thread_start_us":95,"threads_started":1}
I20251026 15:07:41.727927 16342 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:56336:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251026 15:07:41.728103 16342 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:07:41.729683 16632 tablet_service.cc:1505] Processing CreateTablet for tablet 889376c681c848978b041d1c3226c164 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ee50d6d7ac214434836562b556edfd80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 15:07:41.729797 16775 tablet_service.cc:1505] Processing CreateTablet for tablet 889376c681c848978b041d1c3226c164 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ee50d6d7ac214434836562b556edfd80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 15:07:41.729849 16632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 889376c681c848978b041d1c3226c164. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:41.729928 16775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 889376c681c848978b041d1c3226c164. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:41.731348 16866 tablet_bootstrap.cc:492] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768: Bootstrap starting.
I20251026 15:07:41.731411 16487 tablet_service.cc:1505] Processing CreateTablet for tablet 889376c681c848978b041d1c3226c164 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ee50d6d7ac214434836562b556edfd80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 15:07:41.731550 16487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 889376c681c848978b041d1c3226c164. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:41.731789 16866 tablet_bootstrap.cc:654] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:41.732438 16866 tablet_bootstrap.cc:492] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768: No bootstrap required, opened a new log
I20251026 15:07:41.732507 16866 ts_tablet_manager.cc:1403] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:41.732666 16866 raft_consensus.cc:359] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } }
I20251026 15:07:41.732707 16865 tablet_bootstrap.cc:492] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0: Bootstrap starting.
I20251026 15:07:41.732730 16866 raft_consensus.cc:385] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:41.732753 16866 raft_consensus.cc:740] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e705caaa894ffc9f72333bc47e4768, State: Initialized, Role: FOLLOWER
I20251026 15:07:41.732748 16864 tablet_bootstrap.cc:492] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63: Bootstrap starting.
I20251026 15:07:41.732803 16866 consensus_queue.cc:260] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } }
I20251026 15:07:41.732903 16866 ts_tablet_manager.cc:1434] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:41.733038 16872 tablet_replica.cc:442] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } }
I20251026 15:07:41.733098 16865 tablet_bootstrap.cc:654] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:41.733109 16872 tablet_replica.cc:445] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:41.733120 16864 tablet_bootstrap.cc:654] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:41.733798 16864 tablet_bootstrap.cc:492] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63: No bootstrap required, opened a new log
I20251026 15:07:41.733850 16864 ts_tablet_manager.cc:1403] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:41.733953 16865 tablet_bootstrap.cc:492] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0: No bootstrap required, opened a new log
I20251026 15:07:41.733961 16864 raft_consensus.cc:359] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } }
I20251026 15:07:41.734007 16865 ts_tablet_manager.cc:1403] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:41.734018 16864 raft_consensus.cc:385] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:41.734052 16864 raft_consensus.cc:740] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc4759ae84349a696d99e7a75a75c63, State: Initialized, Role: FOLLOWER
I20251026 15:07:41.734109 16864 consensus_queue.cc:260] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } }
I20251026 15:07:41.734165 16865 raft_consensus.cc:359] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } }
I20251026 15:07:41.734205 16864 ts_tablet_manager.cc:1434] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:41.734228 16865 raft_consensus.cc:385] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:41.734253 16865 raft_consensus.cc:740] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3760ef5ebc15404a933e6b314afea2d0, State: Initialized, Role: FOLLOWER
I20251026 15:07:41.734231 16874 tablet_replica.cc:442] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } }
I20251026 15:07:41.734301 16874 tablet_replica.cc:445] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:41.734294 16865 consensus_queue.cc:260] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } }
I20251026 15:07:41.734377 16865 ts_tablet_manager.cc:1434] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:41.734525 16870 tablet_replica.cc:442] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } }
I20251026 15:07:41.734601 16870 tablet_replica.cc:445] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:41.834657 16872 raft_consensus.cc:493] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:07:41.834765 16872 raft_consensus.cc:515] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } }
I20251026 15:07:41.835193 16872 leader_election.cc:290] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bc4759ae84349a696d99e7a75a75c63 (127.14.212.67:38325), 3760ef5ebc15404a933e6b314afea2d0 (127.14.212.65:38713)
I20251026 15:07:41.837669 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.835342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:38713 (local address 127.14.212.66:43861)
1026 15:07:41.835487 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:41.835497 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:41.835543 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:41.835892 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:41.835892 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:41.835894 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:41.836063 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.836064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.836783 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.836784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:41.837361 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.837362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.837449 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.837454 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:07:41.837489 (+ 35us) client_negotiation.cc:240] Negotiation successful
1026 15:07:41.837543 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20251026 15:07:41.837673 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.835342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:38325 (local address 127.14.212.66:36423)
1026 15:07:41.835604 (+ 262us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:41.835616 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:41.835694 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:41.835843 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:41.835844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:41.835846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:41.836071 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.836073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.836738 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.836739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:41.837357 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.837359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.837431 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.837436 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:07:41.837471 (+ 35us) client_negotiation.cc:240] Negotiation successful
1026 15:07:41.837508 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20251026 15:07:41.838138 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.835425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:43861 (local address 127.14.212.65:38713)
1026 15:07:41.835538 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.835540 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.835560 (+ 20us) server_negotiation.cc:408] Connection header received
1026 15:07:41.835596 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.835597 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.835609 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.835743 (+ 134us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:41.836144 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.836697 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.837462 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.837979 (+ 517us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.837989 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.838018 (+ 29us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.838059 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20251026 15:07:41.838156 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.835425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:36423 (local address 127.14.212.67:38325)
1026 15:07:41.835535 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.835538 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.835727 (+ 189us) server_negotiation.cc:408] Connection header received
1026 15:07:41.835762 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.835763 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.835772 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.835825 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:41.836160 (+ 335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.836665 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.837453 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.838013 (+ 560us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.838022 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.838051 (+ 29us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.838092 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20251026 15:07:41.838351 16507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "889376c681c848978b041d1c3226c164" candidate_uuid: "74e705caaa894ffc9f72333bc47e4768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3760ef5ebc15404a933e6b314afea2d0" is_pre_election: true
I20251026 15:07:41.838464 16507 raft_consensus.cc:2468] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e705caaa894ffc9f72333bc47e4768 in term 0.
I20251026 15:07:41.838420 16795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "889376c681c848978b041d1c3226c164" candidate_uuid: "74e705caaa894ffc9f72333bc47e4768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bc4759ae84349a696d99e7a75a75c63" is_pre_election: true
I20251026 15:07:41.838532 16795 raft_consensus.cc:2468] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e705caaa894ffc9f72333bc47e4768 in term 0.
I20251026 15:07:41.838620 16585 leader_election.cc:304] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3760ef5ebc15404a933e6b314afea2d0, 74e705caaa894ffc9f72333bc47e4768; no voters:
I20251026 15:07:41.838752 16872 raft_consensus.cc:2804] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:07:41.838810 16872 raft_consensus.cc:493] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:07:41.838840 16872 raft_consensus.cc:3060] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:41.839506 16872 raft_consensus.cc:515] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } }
I20251026 15:07:41.839632 16872 leader_election.cc:290] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [CANDIDATE]: Term 1 election: Requested vote from peers 8bc4759ae84349a696d99e7a75a75c63 (127.14.212.67:38325), 3760ef5ebc15404a933e6b314afea2d0 (127.14.212.65:38713)
I20251026 15:07:41.839808 16795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "889376c681c848978b041d1c3226c164" candidate_uuid: "74e705caaa894ffc9f72333bc47e4768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bc4759ae84349a696d99e7a75a75c63"
I20251026 15:07:41.839818 16507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "889376c681c848978b041d1c3226c164" candidate_uuid: "74e705caaa894ffc9f72333bc47e4768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3760ef5ebc15404a933e6b314afea2d0"
I20251026 15:07:41.839866 16795 raft_consensus.cc:3060] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:41.839882 16507 raft_consensus.cc:3060] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:41.840449 16507 raft_consensus.cc:2468] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e705caaa894ffc9f72333bc47e4768 in term 1.
I20251026 15:07:41.840495 16795 raft_consensus.cc:2468] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e705caaa894ffc9f72333bc47e4768 in term 1.
I20251026 15:07:41.840667 16582 leader_election.cc:304] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74e705caaa894ffc9f72333bc47e4768, 8bc4759ae84349a696d99e7a75a75c63; no voters:
I20251026 15:07:41.840811 16872 raft_consensus.cc:2804] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:41.840955 16872 raft_consensus.cc:697] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [term 1 LEADER]: Becoming Leader. State: Replica: 74e705caaa894ffc9f72333bc47e4768, State: Running, Role: LEADER
I20251026 15:07:41.841020 16872 consensus_queue.cc:237] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } }
I20251026 15:07:41.841315 16908 tablet_replica.cc:442] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768: TxnStatusTablet state changed. Reason: New leader 74e705caaa894ffc9f72333bc47e4768. Latest consensus state: current_term: 1 leader_uuid: "74e705caaa894ffc9f72333bc47e4768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } }
I20251026 15:07:41.841399 16908 tablet_replica.cc:445] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768: This TxnStatusTablet replica's current role is: LEADER
I20251026 15:07:41.841567 16910 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251026 15:07:41.841667 16910 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251026 15:07:41.841691 16342 catalog_manager.cc:5649] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74e705caaa894ffc9f72333bc47e4768 (127.14.212.66). New cstate: current_term: 1 leader_uuid: "74e705caaa894ffc9f72333bc47e4768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } health_report { overall_health: UNKNOWN } } }
I20251026 15:07:41.850807 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.848050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.14.212.66:46713)
1026 15:07:41.848192 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.848194 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.848203 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:07:41.848228 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.848228 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.848244 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.848301 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:41.848612 (+ 311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.849443 (+ 831us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.850261 (+ 818us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.850414 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.850449 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:41.850547 (+ 98us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:41.850619 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.850727 (+ 108us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.850751 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20251026 15:07:41.850827 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.847809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46713 (local address 127.0.0.1:49932)
1026 15:07:41.847888 (+ 79us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:41.847901 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:41.847963 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:41.848334 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:41.848336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:41.848338 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 15:07:41.848543 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.848545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.849523 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.849524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:41.850185 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.850187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.850267 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.850286 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 15:07:41.850634 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 15:07:41.850646 (+ 12us) client_negotiation.cc:769] Sending connection context
1026 15:07:41.850703 (+ 57us) client_negotiation.cc:240] Negotiation successful
1026 15:07:41.850753 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20251026 15:07:41.851902 16795 raft_consensus.cc:1275] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63 [term 1 FOLLOWER]: Refusing update from remote peer 74e705caaa894ffc9f72333bc47e4768: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:41.852070 16507 raft_consensus.cc:1275] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0 [term 1 FOLLOWER]: Refusing update from remote peer 74e705caaa894ffc9f72333bc47e4768: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:41.852283 16872 consensus_queue.cc:1048] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:41.852380 16908 consensus_queue.cc:1048] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:41.853061 16870 tablet_replica.cc:442] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0: TxnStatusTablet state changed. Reason: New leader 74e705caaa894ffc9f72333bc47e4768. Latest consensus state: current_term: 1 leader_uuid: "74e705caaa894ffc9f72333bc47e4768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } }
I20251026 15:07:41.853149 16870 tablet_replica.cc:445] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:41.853369 16874 tablet_replica.cc:442] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63: TxnStatusTablet state changed. Reason: New leader 74e705caaa894ffc9f72333bc47e4768. Latest consensus state: current_term: 1 leader_uuid: "74e705caaa894ffc9f72333bc47e4768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } }
I20251026 15:07:41.853451 16874 tablet_replica.cc:445] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:41.853860 16914 mvcc.cc:204] Tried to move back new op lower bound from 7215068208543502336 to 7215068208501370880. Current Snapshot: MvccSnapshot[applied={T|T < 7215068208543502336}]
I20251026 15:07:41.853933 16872 tablet_replica.cc:442] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74e705caaa894ffc9f72333bc47e4768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } }
I20251026 15:07:41.853999 16872 tablet_replica.cc:445] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768: This TxnStatusTablet replica's current role is: LEADER
I20251026 15:07:41.854007 16870 tablet_replica.cc:442] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74e705caaa894ffc9f72333bc47e4768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } }
I20251026 15:07:41.854079 16870 tablet_replica.cc:445] T 889376c681c848978b041d1c3226c164 P 3760ef5ebc15404a933e6b314afea2d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:41.854074 16872 tablet_replica.cc:442] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74e705caaa894ffc9f72333bc47e4768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } }
I20251026 15:07:41.854126 16872 tablet_replica.cc:445] T 889376c681c848978b041d1c3226c164 P 74e705caaa894ffc9f72333bc47e4768: This TxnStatusTablet replica's current role is: LEADER
I20251026 15:07:41.854738 16874 tablet_replica.cc:442] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74e705caaa894ffc9f72333bc47e4768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 } } peers { permanent_uuid: "8bc4759ae84349a696d99e7a75a75c63" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38325 } } peers { permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 } } }
I20251026 15:07:41.854810 16874 tablet_replica.cc:445] T 889376c681c848978b041d1c3226c164 P 8bc4759ae84349a696d99e7a75a75c63: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:41.855348 16917 mvcc.cc:204] Tried to move back new op lower bound from 7215068208543502336 to 7215068208501370880. Current Snapshot: MvccSnapshot[applied={T|T < 7215068208543502336}]
I20251026 15:07:41.860209 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.857922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:50885 (local address 127.14.212.66:46713)
1026 15:07:41.857980 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.857982 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.858111 (+ 129us) server_negotiation.cc:408] Connection header received
1026 15:07:41.858175 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.858176 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.858184 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.858231 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:41.858579 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.859048 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.859855 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.859985 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.860002 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:41.860021 (+ 19us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:41.860066 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.860148 (+ 82us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.860164 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251026 15:07:41.860401 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.857869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46713 (local address 127.14.212.67:50885)
1026 15:07:41.858041 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:41.858053 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:41.858116 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:41.858255 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:41.858256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:41.858258 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 15:07:41.858513 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.858515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.859125 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.859126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:41.859777 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.859779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.859852 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.859870 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 15:07:41.860082 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 15:07:41.860083 (+ 1us) client_negotiation.cc:769] Sending connection context
1026 15:07:41.860132 (+ 49us) client_negotiation.cc:240] Negotiation successful
1026 15:07:41.860178 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20251026 15:07:41.862454 16652 raft_consensus.cc:1275] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768 [term 1 FOLLOWER]: Refusing update from remote peer 8bc4759ae84349a696d99e7a75a75c63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:41.862499 16507 raft_consensus.cc:1275] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0 [term 1 FOLLOWER]: Refusing update from remote peer 8bc4759ae84349a696d99e7a75a75c63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:41.862658 16874 consensus_queue.cc:1048] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74e705caaa894ffc9f72333bc47e4768" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:41.862758 16871 consensus_queue.cc:1048] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3760ef5ebc15404a933e6b314afea2d0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:41.871147 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.868325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:38325 (local address 127.14.212.66:39455)
1026 15:07:41.868607 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:41.868618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:41.868692 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:41.868831 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:41.868832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:41.868841 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 15:07:41.869136 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.869137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.869991 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.869992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:41.870633 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:41.870636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.870714 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.870732 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 15:07:41.871012 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 15:07:41.871013 (+ 1us) client_negotiation.cc:769] Sending connection context
1026 15:07:41.871048 (+ 35us) client_negotiation.cc:240] Negotiation successful
1026 15:07:41.871085 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":70,"threads_started":1}
I20251026 15:07:41.871141 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.868490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:39455 (local address 127.14.212.67:38325)
1026 15:07:41.868530 (+ 40us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.868533 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.868696 (+ 163us) server_negotiation.cc:408] Connection header received
1026 15:07:41.868763 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.868764 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.868774 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.868825 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:41.869196 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.869909 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.870707 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.870887 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.870909 (+ 22us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:41.870945 (+ 36us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:41.871007 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.871050 (+ 43us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.871073 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20251026 15:07:41.880085 16342 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56332:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:07:41.880203 16342 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56332:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:07:41.880864 16342 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 7088cfbe1e3e4e708cc7fd8afa3463ca: Sending DeleteTablet for 3 replicas of tablet 1c9e804c9a63493eb3f71807e52f037c
I20251026 15:07:41.881131 16775 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c9e804c9a63493eb3f71807e52f037c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:42704
I20251026 15:07:41.881186 16632 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c9e804c9a63493eb3f71807e52f037c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:49920
I20251026 15:07:41.881191 16487 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c9e804c9a63493eb3f71807e52f037c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:42058
I20251026 15:07:41.881379 16933 tablet_replica.cc:333] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0: stopping tablet replica
I20251026 15:07:41.881435 16933 raft_consensus.cc:2243] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:07:41.881469 16933 raft_consensus.cc:2272] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:41.881723 16933 ts_tablet_manager.cc:1916] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:41.882043 16932 tablet_replica.cc:333] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63: stopping tablet replica
I20251026 15:07:41.882138 16932 raft_consensus.cc:2243] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:07:41.882237 16932 raft_consensus.cc:2272] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:41.882501 16934 tablet_replica.cc:333] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768: stopping tablet replica
I20251026 15:07:41.882529 16932 ts_tablet_manager.cc:1916] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:41.882629 16934 raft_consensus.cc:2243] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:07:41.882711 16934 raft_consensus.cc:2272] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:41.883011 16934 ts_tablet_manager.cc:1916] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:41.883477 16933 ts_tablet_manager.cc:1929] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:07:41.883543 16933 log.cc:1199] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-0/wal/wals/1c9e804c9a63493eb3f71807e52f037c
I20251026 15:07:41.883772 16933 ts_tablet_manager.cc:1950] T 1c9e804c9a63493eb3f71807e52f037c P 3760ef5ebc15404a933e6b314afea2d0: Deleting consensus metadata
I20251026 15:07:41.884099 16329 catalog_manager.cc:4985] TS 3760ef5ebc15404a933e6b314afea2d0 (127.14.212.65:38713): tablet 1c9e804c9a63493eb3f71807e52f037c (table test-table [id=97524128c36b424cada08934f51e96c6]) successfully deleted
I20251026 15:07:41.884673 16934 ts_tablet_manager.cc:1929] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:07:41.884742 16934 log.cc:1199] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/1c9e804c9a63493eb3f71807e52f037c
I20251026 15:07:41.884938 16934 ts_tablet_manager.cc:1950] T 1c9e804c9a63493eb3f71807e52f037c P 74e705caaa894ffc9f72333bc47e4768: Deleting consensus metadata
I20251026 15:07:41.885262 16327 catalog_manager.cc:4985] TS 74e705caaa894ffc9f72333bc47e4768 (127.14.212.66:46713): tablet 1c9e804c9a63493eb3f71807e52f037c (table test-table [id=97524128c36b424cada08934f51e96c6]) successfully deleted
I20251026 15:07:41.886297 16932 ts_tablet_manager.cc:1929] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:07:41.886365 16932 log.cc:1199] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/1c9e804c9a63493eb3f71807e52f037c
I20251026 15:07:41.886615 16932 ts_tablet_manager.cc:1950] T 1c9e804c9a63493eb3f71807e52f037c P 8bc4759ae84349a696d99e7a75a75c63: Deleting consensus metadata
I20251026 15:07:41.886916 16326 catalog_manager.cc:4985] TS 8bc4759ae84349a696d99e7a75a75c63 (127.14.212.67:38325): tablet 1c9e804c9a63493eb3f71807e52f037c (table test-table [id=97524128c36b424cada08934f51e96c6]) successfully deleted
Oct 26 15:07:41 dist-test-slave-8pgz krb5kdc[16292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491261, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.212.65@KRBTEST.COM
I20251026 15:07:41.891346 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.882075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.14.212.65:38713)
1026 15:07:41.882122 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.882125 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.882925 (+ 800us) server_negotiation.cc:408] Connection header received
1026 15:07:41.883031 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.883032 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.883042 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.883072 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:41.884287 (+ 1215us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.885026 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.885849 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.886015 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.888596 (+ 2581us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:41.888605 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:41.888605 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:41.888612 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:41.890256 (+ 1644us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:41.890681 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:41.890684 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:41.890684 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:41.890725 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:41.890880 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:41.890880 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:41.890880 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:41.891001 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:41.891114 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.891210 (+ 96us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.891284 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20251026 15:07:41.891562 16549 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42068
I20251026 15:07:41.898123 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:41.892439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.14.212.126:36573)
1026 15:07:41.892483 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:41.892485 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:41.892577 (+ 92us) server_negotiation.cc:408] Connection header received
1026 15:07:41.892607 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:41.892608 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:41.892623 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:41.892683 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:41.893284 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.894073 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:41.894628 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:41.894803 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:41.895696 (+ 893us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:41.895700 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:41.895700 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:41.895707 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:41.897202 (+ 1495us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:41.897500 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:41.897501 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:41.897501 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:41.897537 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:41.897656 (+ 119us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:41.897656 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:41.897657 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:41.897760 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:41.897860 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:41.897930 (+ 70us) server_negotiation.cc:300] Negotiation successful
1026 15:07:41.898024 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20251026 15:07:41.898375 16342 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56338
I20251026 15:07:41.898854 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 16420
I20251026 15:07:41.904984 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 16566
I20251026 15:07:41.910142 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 16709
I20251026 15:07:41.915743 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 16309
2025-10-26T15:07:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3702 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:07:41 dist-test-slave-8pgz krb5kdc[16941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:07:41 dist-test-slave-8pgz krb5kdc[16941](info): set up 2 sockets
Oct 26 15:07:41 dist-test-slave-8pgz krb5kdc[16941](info): commencing operation
krb5kdc: starting...
W20251026 15:07:43.911815 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.975s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:07:43 dist-test-slave-8pgz krb5kdc[16941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:07:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:07:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:44.047247 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:42195
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:42195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:44.127559 16957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:44.127763 16957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:44.127800 16957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:44.129220 16957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:07:44.129269 16957 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251026 15:07:44.129297 16957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:44.129319 16957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:44.129343 16957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:07:44.129374 16957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:07:44.131039 16957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:42195
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:42195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:44.131276 16957 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:44.131464 16957 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:44.133863 16962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:44.133911 16965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:44.133863 16963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:44.134114 16957 server_base.cc:1047] running on GCE node
I20251026 15:07:44.134389 16957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:44.134677 16957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:44.135857 16957 hybrid_clock.cc:648] HybridClock initialized: now 1761491264135844 us; error 31 us; skew 500 ppm
Oct 26 15:07:44 dist-test-slave-8pgz krb5kdc[16941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491264, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:44.138252 16957 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:07:44.138789 16957 webserver.cc:492] Webserver started at http://127.14.212.126:38793/ using document root <none> and password file <none>
I20251026 15:07:44.138952 16957 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:44.138983 16957 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:44.139063 16957 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:44.139956 16957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "be14f780bfe74bdc98766fe9ab0ab606"
format_stamp: "Formatted at 2025-10-26 15:07:44 on dist-test-slave-8pgz"
server_key: "f3dddf3291552b899af9279de46b1e00"
server_key_iv: "d75c12e292140b5eab7317a1b4665ad4"
server_key_version: "encryptionkey@0"
I20251026 15:07:44.140225 16957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "be14f780bfe74bdc98766fe9ab0ab606"
format_stamp: "Formatted at 2025-10-26 15:07:44 on dist-test-slave-8pgz"
server_key: "f3dddf3291552b899af9279de46b1e00"
server_key_iv: "d75c12e292140b5eab7317a1b4665ad4"
server_key_version: "encryptionkey@0"
I20251026 15:07:44.141368 16957 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251026 15:07:44.142061 16972 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:44.142239 16957 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:44.142304 16957 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "be14f780bfe74bdc98766fe9ab0ab606"
format_stamp: "Formatted at 2025-10-26 15:07:44 on dist-test-slave-8pgz"
server_key: "f3dddf3291552b899af9279de46b1e00"
server_key_iv: "d75c12e292140b5eab7317a1b4665ad4"
server_key_version: "encryptionkey@0"
I20251026 15:07:44.142375 16957 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:44.155838 16957 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:44.158200 16957 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:44.158324 16957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:44.164306 16957 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:42195
I20251026 15:07:44.164348 17035 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:42195 every 8 connection(s)
I20251026 15:07:44.164670 16957 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:07:44.165269 17036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:44.167716 17036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606: Bootstrap starting.
I20251026 15:07:44.168279 17036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:44.168511 17036 log.cc:826] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:44.169205 17036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606: No bootstrap required, opened a new log
I20251026 15:07:44.170356 17036 raft_consensus.cc:359] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be14f780bfe74bdc98766fe9ab0ab606" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42195 } }
I20251026 15:07:44.170506 17036 raft_consensus.cc:385] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:44.170542 17036 raft_consensus.cc:740] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be14f780bfe74bdc98766fe9ab0ab606, State: Initialized, Role: FOLLOWER
I20251026 15:07:44.170666 17036 consensus_queue.cc:260] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be14f780bfe74bdc98766fe9ab0ab606" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42195 } }
I20251026 15:07:44.170723 17036 raft_consensus.cc:399] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:07:44.170754 17036 raft_consensus.cc:493] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:07:44.170799 17036 raft_consensus.cc:3060] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:44.171315 17036 raft_consensus.cc:515] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be14f780bfe74bdc98766fe9ab0ab606" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42195 } }
I20251026 15:07:44.171414 17036 leader_election.cc:304] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be14f780bfe74bdc98766fe9ab0ab606; no voters:
I20251026 15:07:44.171550 17036 leader_election.cc:290] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:07:44.171613 17039 raft_consensus.cc:2804] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:44.171772 17036 sys_catalog.cc:565] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:07:44.171780 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 16957
I20251026 15:07:44.171869 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:07:44.171898 17039 raft_consensus.cc:697] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [term 1 LEADER]: Becoming Leader. State: Replica: be14f780bfe74bdc98766fe9ab0ab606, State: Running, Role: LEADER
I20251026 15:07:44.171988 17039 consensus_queue.cc:237] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be14f780bfe74bdc98766fe9ab0ab606" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42195 } }
I20251026 15:07:44.172102 15185 external_mini_cluster.cc:1468] Setting key d9f7f518bb7f01a3b0d30db7ce41342a
I20251026 15:07:44.172304 17039 sys_catalog.cc:455] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be14f780bfe74bdc98766fe9ab0ab606. Latest consensus state: current_term: 1 leader_uuid: "be14f780bfe74bdc98766fe9ab0ab606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be14f780bfe74bdc98766fe9ab0ab606" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42195 } } }
I20251026 15:07:44.172369 17039 sys_catalog.cc:458] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [sys.catalog]: This master's current role is: LEADER
I20251026 15:07:44.172609 17046 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:07:44.172881 17040 sys_catalog.cc:455] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be14f780bfe74bdc98766fe9ab0ab606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be14f780bfe74bdc98766fe9ab0ab606" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42195 } } }
I20251026 15:07:44.172955 17040 sys_catalog.cc:458] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606 [sys.catalog]: This master's current role is: LEADER
I20251026 15:07:44.173067 17046 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:07:44.174484 17056 mvcc.cc:204] Tried to move back new op lower bound from 7215068218056470528 to 7215068218048815104. Current Snapshot: MvccSnapshot[applied={T|T < 7215068218056470528}]
I20251026 15:07:44.174862 17046 catalog_manager.cc:1357] Generated new cluster ID: 960ecfd2fda04136883cc2cca7674104
I20251026 15:07:44.174916 17046 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 26 15:07:44 dist-test-slave-8pgz krb5kdc[16941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:44.181758 17046 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:07:44.182009 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.172591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.14.212.126:42195)
1026 15:07:44.172966 (+ 375us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:44.172976 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:44.172998 (+ 22us) server_negotiation.cc:408] Connection header received
1026 15:07:44.173101 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:44.173107 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:44.173158 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:44.173282 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:44.174279 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.175490 (+ 1211us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.176054 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.176636 (+ 582us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.178368 (+ 1732us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:44.178374 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:44.178374 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:44.178418 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:44.180578 (+ 2160us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.180974 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.180976 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.180977 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.181038 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.181198 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.181199 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.181199 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.181406 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:44.181543 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:44.181714 (+ 171us) server_negotiation.cc:300] Negotiation successful
1026 15:07:44.181850 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":71,"threads_started":1}
I20251026 15:07:44.182530 17046 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:07:44.187528 17046 catalog_manager.cc:6022] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606: Generated new TSK 0
I20251026 15:07:44.187676 17046 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:44.233444 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:42195
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:44.314555 17064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:44.314738 17064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:44.314767 17064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:44.316198 17064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:44.316254 17064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:44.316277 17064 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251026 15:07:44.316291 17064 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 15:07:44.316309 17064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:07:44.318048 17064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.212.126:42195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:44.318296 17064 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:44.318529 17064 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:44.321182 17069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:44.321230 17070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:44.321195 17072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:44.321573 17064 server_base.cc:1047] running on GCE node
I20251026 15:07:44.321753 17064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:44.321965 17064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:44.323119 17064 hybrid_clock.cc:648] HybridClock initialized: now 1761491264323102 us; error 34 us; skew 500 ppm
Oct 26 15:07:44 dist-test-slave-8pgz krb5kdc[16941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491264, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:44.325577 17064 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:07:44.326136 17064 webserver.cc:492] Webserver started at http://127.14.212.65:43737/ using document root <none> and password file <none>
I20251026 15:07:44.326331 17064 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:44.326376 17064 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:44.326538 17064 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:44.327612 17064 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "33f9f52d8faf498e8ddb1f470bce5bd2"
format_stamp: "Formatted at 2025-10-26 15:07:44 on dist-test-slave-8pgz"
server_key: "753ded63eba83ba2a0bd018ac12ebd1d"
server_key_iv: "fb726202361b52653374b1a45e121f6e"
server_key_version: "encryptionkey@0"
I20251026 15:07:44.327946 17064 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "33f9f52d8faf498e8ddb1f470bce5bd2"
format_stamp: "Formatted at 2025-10-26 15:07:44 on dist-test-slave-8pgz"
server_key: "753ded63eba83ba2a0bd018ac12ebd1d"
server_key_iv: "fb726202361b52653374b1a45e121f6e"
server_key_version: "encryptionkey@0"
I20251026 15:07:44.329263 17064 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:07:44.329968 17079 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:44.330164 17064 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:44.330242 17064 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "33f9f52d8faf498e8ddb1f470bce5bd2"
format_stamp: "Formatted at 2025-10-26 15:07:44 on dist-test-slave-8pgz"
server_key: "753ded63eba83ba2a0bd018ac12ebd1d"
server_key_iv: "fb726202361b52653374b1a45e121f6e"
server_key_version: "encryptionkey@0"
I20251026 15:07:44.330308 17064 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:44.340683 17064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:44.342815 17064 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:44.342948 17064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:44.343472 17064 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:44.343513 17064 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:44.343551 17064 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:44.343582 17064 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:44.350301 17064 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:38959
I20251026 15:07:44.350345 17195 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:38959 every 8 connection(s)
Oct 26 15:07:44 dist-test-slave-8pgz krb5kdc[16941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491264, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:44.350715 17064 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:07:44.353493 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.343933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:45597 (local address 127.14.212.126:42195)
1026 15:07:44.343999 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:44.344002 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:44.344172 (+ 170us) server_negotiation.cc:408] Connection header received
1026 15:07:44.344677 (+ 505us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:44.344679 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:44.344698 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:44.344789 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:44.345927 (+ 1138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.346885 (+ 958us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.347869 (+ 984us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.348113 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.351209 (+ 3096us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:44.351213 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:44.351214 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:44.351220 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:44.352467 (+ 1247us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.352821 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.352822 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.352822 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.352840 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.352982 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.352982 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.352982 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.353077 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:44.353132 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:44.353353 (+ 221us) server_negotiation.cc:300] Negotiation successful
1026 15:07:44.353429 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251026 15:07:44.353602 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.343881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:42195 (local address 127.14.212.65:45597)
1026 15:07:44.344053 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:44.344083 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:44.344582 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:44.344856 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:44.344860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:44.345307 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:44.345799 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.345805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.347009 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.347010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:44.347739 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.347743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.347940 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.348562 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:44.348585 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:44.351118 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:44.352548 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:44.352553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:44.352556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:44.352756 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:44.352886 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:44.352886 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:44.352887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:44.352941 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:44.353121 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:44.353123 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:44.353248 (+ 125us) client_negotiation.cc:769] Sending connection context
1026 15:07:44.353338 (+ 90us) client_negotiation.cc:240] Negotiation successful
1026 15:07:44.353436 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":99,"threads_started":1}
I20251026 15:07:44.354156 17196 heartbeater.cc:344] Connected to a master server at 127.14.212.126:42195
I20251026 15:07:44.354238 17196 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:44.354457 17196 heartbeater.cc:507] Master 127.14.212.126:42195 requested a full tablet report, sending...
I20251026 15:07:44.354949 16988 ts_manager.cc:194] Registered new tserver with Master: 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65:38959)
I20251026 15:07:44.355813 16988 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:45597
I20251026 15:07:44.357999 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 17064
I20251026 15:07:44.358109 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:07:44.358318 15185 external_mini_cluster.cc:1468] Setting key 5f17c749c18211888a972ba0eb049737
I20251026 15:07:44.360405 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.355197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:42195 (local address 127.14.212.65:39957)
1026 15:07:44.355328 (+ 131us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:44.355337 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:44.355387 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:44.355514 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:44.355514 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:44.355633 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:44.355832 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.355833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.356562 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.356563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:44.356969 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.356971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.357041 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.357477 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:44.357481 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:44.357752 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:44.359617 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:44.359617 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:44.359618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:44.359775 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:44.359941 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:44.359942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:44.359942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:44.359964 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:44.360180 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:44.360180 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:44.360211 (+ 31us) client_negotiation.cc:769] Sending connection context
1026 15:07:44.360257 (+ 46us) client_negotiation.cc:240] Negotiation successful
1026 15:07:44.360328 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20251026 15:07:44.360515 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.355272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:39957 (local address 127.14.212.126:42195)
1026 15:07:44.355314 (+ 42us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:44.355316 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:44.355377 (+ 61us) server_negotiation.cc:408] Connection header received
1026 15:07:44.355440 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:44.355441 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:44.355450 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:44.355512 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:44.355904 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.356466 (+ 562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.357041 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.357142 (+ 101us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.357832 (+ 690us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:44.357835 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:44.357835 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:44.357840 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:44.359531 (+ 1691us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.359844 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.359845 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.359845 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.359870 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.360008 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.360009 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.360009 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.360119 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:44.360196 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:44.360282 (+ 86us) server_negotiation.cc:300] Negotiation successful
1026 15:07:44.360380 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:44.405601 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:42195
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:44.484005 17207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:44.484196 17207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:44.484233 17207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:44.485647 17207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:44.485700 17207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:44.485735 17207 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251026 15:07:44.485754 17207 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 15:07:44.485782 17207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:07:44.487422 17207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.212.126:42195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:44.487648 17207 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:44.487845 17207 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:44.490243 17212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:44.490291 17215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:44.490366 17213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:44.490572 17207 server_base.cc:1047] running on GCE node
I20251026 15:07:44.490775 17207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:44.490957 17207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:44.492120 17207 hybrid_clock.cc:648] HybridClock initialized: now 1761491264492106 us; error 37 us; skew 500 ppm
Oct 26 15:07:44 dist-test-slave-8pgz krb5kdc[16941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491264, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:44.494782 17207 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:07:44.495327 17207 webserver.cc:492] Webserver started at http://127.14.212.66:40121/ using document root <none> and password file <none>
I20251026 15:07:44.495509 17207 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:44.495550 17207 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:44.495664 17207 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:44.496577 17207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "bf317d62a3f0473fa8e54db7d365221c"
format_stamp: "Formatted at 2025-10-26 15:07:44 on dist-test-slave-8pgz"
server_key: "6abc7e54528884afe631e7c609d55984"
server_key_iv: "7e87f85269df40885cbe6d176413f3c0"
server_key_version: "encryptionkey@0"
I20251026 15:07:44.496876 17207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "bf317d62a3f0473fa8e54db7d365221c"
format_stamp: "Formatted at 2025-10-26 15:07:44 on dist-test-slave-8pgz"
server_key: "6abc7e54528884afe631e7c609d55984"
server_key_iv: "7e87f85269df40885cbe6d176413f3c0"
server_key_version: "encryptionkey@0"
I20251026 15:07:44.498067 17207 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:07:44.498777 17222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:44.498936 17207 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:44.499009 17207 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "bf317d62a3f0473fa8e54db7d365221c"
format_stamp: "Formatted at 2025-10-26 15:07:44 on dist-test-slave-8pgz"
server_key: "6abc7e54528884afe631e7c609d55984"
server_key_iv: "7e87f85269df40885cbe6d176413f3c0"
server_key_version: "encryptionkey@0"
I20251026 15:07:44.499069 17207 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:44.513835 17207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:44.515762 17207 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:44.515903 17207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:44.516438 17207 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:44.516480 17207 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:44.516516 17207 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:44.516544 17207 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 15:07:44 dist-test-slave-8pgz krb5kdc[16941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491264, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:44.523723 17207 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:44241
I20251026 15:07:44.523823 17338 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:44241 every 8 connection(s)
I20251026 15:07:44.524104 17207 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:07:44.525207 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.516844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:33243 (local address 127.14.212.126:42195)
1026 15:07:44.516918 (+ 74us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:44.516921 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:44.517046 (+ 125us) server_negotiation.cc:408] Connection header received
1026 15:07:44.517541 (+ 495us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:44.517543 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:44.517558 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:44.517638 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:44.518529 (+ 891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.519365 (+ 836us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.520046 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.520237 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.522226 (+ 1989us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:44.522230 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:44.522231 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:44.522238 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:44.524101 (+ 1863us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.524514 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.524516 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.524516 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.524547 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.524715 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.524716 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.524716 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.524806 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:44.524853 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:44.525068 (+ 215us) server_negotiation.cc:300] Negotiation successful
1026 15:07:44.525146 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251026 15:07:44.525324 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.516783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:42195 (local address 127.14.212.66:33243)
1026 15:07:44.516928 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:44.516959 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:44.517442 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:44.517691 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:44.517696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:44.518094 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:44.518433 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.518440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.519476 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.519477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:44.519952 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.519955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.520084 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.520740 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:44.520756 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:44.522119 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:44.524182 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:44.524187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:44.524190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:44.524441 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:44.524619 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:44.524620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:44.524620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:44.524676 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:44.524848 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:44.524850 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:44.524963 (+ 113us) client_negotiation.cc:769] Sending connection context
1026 15:07:44.525046 (+ 83us) client_negotiation.cc:240] Negotiation successful
1026 15:07:44.525152 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
I20251026 15:07:44.525821 17339 heartbeater.cc:344] Connected to a master server at 127.14.212.126:42195
I20251026 15:07:44.525900 17339 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:44.526098 17339 heartbeater.cc:507] Master 127.14.212.126:42195 requested a full tablet report, sending...
I20251026 15:07:44.526751 16989 ts_manager.cc:194] Registered new tserver with Master: bf317d62a3f0473fa8e54db7d365221c (127.14.212.66:44241)
I20251026 15:07:44.527308 16989 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:33243
I20251026 15:07:44.530123 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 17207
I20251026 15:07:44.530215 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:07:44.530417 15185 external_mini_cluster.cc:1468] Setting key 4096547e78a2ae85cc1bcdec23ff73ae
I20251026 15:07:44.532085 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.526855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:34177 (local address 127.14.212.126:42195)
1026 15:07:44.526893 (+ 38us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:44.526895 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:44.527022 (+ 127us) server_negotiation.cc:408] Connection header received
1026 15:07:44.527097 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:44.527098 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:44.527105 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:44.527152 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:44.527593 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.528071 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.528605 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.528719 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.529497 (+ 778us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:44.529500 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:44.529501 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:44.529507 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:44.531114 (+ 1607us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.531444 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.531445 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.531445 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.531471 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.531606 (+ 135us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.531607 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.531607 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.531720 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:44.531787 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:44.531913 (+ 126us) server_negotiation.cc:300] Negotiation successful
1026 15:07:44.531994 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20251026 15:07:44.532099 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.526801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:42195 (local address 127.14.212.66:34177)
1026 15:07:44.526946 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:44.526958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:44.527026 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:44.527166 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:44.527167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:44.527329 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:44.527528 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.527531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.528152 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.528153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:44.528550 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.528551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.528607 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.529120 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:44.529124 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:44.529370 (+ 246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:44.531206 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:44.531207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:44.531208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:44.531374 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:44.531525 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:44.531526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:44.531526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:44.531555 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:44.531776 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:44.531777 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:44.531828 (+ 51us) client_negotiation.cc:769] Sending connection context
1026 15:07:44.531887 (+ 59us) client_negotiation.cc:240] Negotiation successful
1026 15:07:44.531995 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:44.577666 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:42195
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:44.658135 17350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:44.658335 17350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:44.658363 17350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:44.659865 17350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:44.659919 17350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:44.659942 17350 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251026 15:07:44.659956 17350 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 15:07:44.659981 17350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:07:44.661548 17350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.212.126:42195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:44.661767 17350 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:44.661939 17350 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:44.664750 17358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:44.664775 17356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:44.664824 17355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:44.664919 17350 server_base.cc:1047] running on GCE node
I20251026 15:07:44.665077 17350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:44.665283 17350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:44.666433 17350 hybrid_clock.cc:648] HybridClock initialized: now 1761491264666411 us; error 40 us; skew 500 ppm
Oct 26 15:07:44 dist-test-slave-8pgz krb5kdc[16941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491264, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:44.668860 17350 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:07:44.669422 17350 webserver.cc:492] Webserver started at http://127.14.212.67:46099/ using document root <none> and password file <none>
I20251026 15:07:44.669608 17350 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:44.669643 17350 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:44.669765 17350 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:44.670763 17350 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "0e28fc72acb34dedbad811e3b12702b6"
format_stamp: "Formatted at 2025-10-26 15:07:44 on dist-test-slave-8pgz"
server_key: "1eaccf5d7687c61a61e4d460781f1066"
server_key_iv: "a17c09fbd488981611bb1a9551e8625d"
server_key_version: "encryptionkey@0"
I20251026 15:07:44.671073 17350 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "0e28fc72acb34dedbad811e3b12702b6"
format_stamp: "Formatted at 2025-10-26 15:07:44 on dist-test-slave-8pgz"
server_key: "1eaccf5d7687c61a61e4d460781f1066"
server_key_iv: "a17c09fbd488981611bb1a9551e8625d"
server_key_version: "encryptionkey@0"
I20251026 15:07:44.672322 17350 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:07:44.673054 17365 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:44.673239 17350 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:44.673336 17350 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "0e28fc72acb34dedbad811e3b12702b6"
format_stamp: "Formatted at 2025-10-26 15:07:44 on dist-test-slave-8pgz"
server_key: "1eaccf5d7687c61a61e4d460781f1066"
server_key_iv: "a17c09fbd488981611bb1a9551e8625d"
server_key_version: "encryptionkey@0"
I20251026 15:07:44.673398 17350 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:44.693039 17350 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:44.695210 17350 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:44.695353 17350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:44.695887 17350 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:44.695932 17350 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:44.695968 17350 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:44.695989 17350 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:44.702644 17350 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:41061
I20251026 15:07:44.702710 17480 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:41061 every 8 connection(s)
Oct 26 15:07:44 dist-test-slave-8pgz krb5kdc[16941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491264, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:44.703018 17350 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:07:44.705535 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.696342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:59787 (local address 127.14.212.126:42195)
1026 15:07:44.696408 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:44.696411 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:44.696596 (+ 185us) server_negotiation.cc:408] Connection header received
1026 15:07:44.697069 (+ 473us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:44.697070 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:44.697086 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:44.697188 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:44.698455 (+ 1267us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.699297 (+ 842us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.700130 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.700358 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.703320 (+ 2962us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:44.703325 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:44.703326 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:44.703333 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:44.704564 (+ 1231us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.704898 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.704899 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.704899 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.704916 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.705078 (+ 162us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.705078 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.705078 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.705159 (+ 81us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:44.705206 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:44.705410 (+ 204us) server_negotiation.cc:300] Negotiation successful
1026 15:07:44.705474 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251026 15:07:44.705628 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.696299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:42195 (local address 127.14.212.67:59787)
1026 15:07:44.696479 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:44.696508 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:44.696979 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:44.697232 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:44.697236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:44.697699 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:44.698301 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.698307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.699387 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.699389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:44.700037 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.700040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.700195 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.700808 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:44.700827 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:44.703230 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:44.704641 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:44.704646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:44.704649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:44.704841 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:44.704976 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:44.704986 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:44.704987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:44.705035 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:44.705200 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:44.705203 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:44.705311 (+ 108us) client_negotiation.cc:769] Sending connection context
1026 15:07:44.705390 (+ 79us) client_negotiation.cc:240] Negotiation successful
1026 15:07:44.705485 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20251026 15:07:44.706144 17481 heartbeater.cc:344] Connected to a master server at 127.14.212.126:42195
I20251026 15:07:44.706230 17481 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:44.706471 17481 heartbeater.cc:507] Master 127.14.212.126:42195 requested a full tablet report, sending...
I20251026 15:07:44.707002 16988 ts_manager.cc:194] Registered new tserver with Master: 0e28fc72acb34dedbad811e3b12702b6 (127.14.212.67:41061)
I20251026 15:07:44.707535 16988 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:59787
I20251026 15:07:44.711529 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.707247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:33135 (local address 127.14.212.126:42195)
1026 15:07:44.707318 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:44.707320 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:44.707388 (+ 68us) server_negotiation.cc:408] Connection header received
1026 15:07:44.707437 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:44.707438 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:44.707447 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:44.707493 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:44.707910 (+ 417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.708362 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.708852 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.708963 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.709575 (+ 612us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:44.709577 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:44.709577 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:44.709582 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:44.710857 (+ 1275us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.711114 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.711115 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.711115 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.711131 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.711224 (+ 93us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.711224 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.711225 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.711293 (+ 68us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:44.711336 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:44.711418 (+ 82us) server_negotiation.cc:300] Negotiation successful
1026 15:07:44.711471 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251026 15:07:44.711551 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.707200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:42195 (local address 127.14.212.67:33135)
1026 15:07:44.707318 (+ 118us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:44.707331 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:44.707384 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:44.707500 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:44.707501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:44.707616 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:44.707831 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.707833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.708428 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.708429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:44.708793 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.708794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.708851 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.709280 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:44.709284 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:44.709507 (+ 223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:44.710925 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:44.710926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:44.710927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:44.711058 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:44.711171 (+ 113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:44.711171 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:44.711171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:44.711193 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:44.711331 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:44.711331 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:44.711361 (+ 30us) client_negotiation.cc:769] Sending connection context
1026 15:07:44.711405 (+ 44us) client_negotiation.cc:240] Negotiation successful
1026 15:07:44.711472 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20251026 15:07:44.712806 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 17350
I20251026 15:07:44.712884 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:07:44.713047 15185 external_mini_cluster.cc:1468] Setting key 3486e5775cadec304bcefe4a52353a4c
I20251026 15:07:44.713759 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 26 15:07:44 dist-test-slave-8pgz krb5kdc[16941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 26 15:07:44 dist-test-slave-8pgz krb5kdc[16941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:44.734314 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.727658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.14.212.126:42195)
1026 15:07:44.727719 (+ 61us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:44.727721 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:44.727756 (+ 35us) server_negotiation.cc:408] Connection header received
1026 15:07:44.727847 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:44.727848 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:44.727860 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:44.727915 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:44.728621 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.729105 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.729644 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.729835 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.731854 (+ 2019us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:44.731858 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:44.731858 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:44.731864 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:44.733266 (+ 1402us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.733590 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.733591 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.733591 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.733616 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:44.733808 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:44.733809 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:44.733809 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:44.733900 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:44.734007 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:44.734104 (+ 97us) server_negotiation.cc:300] Negotiation successful
1026 15:07:44.734187 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251026 15:07:44.736073 16988 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:07:44.737442 16988 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:07:44.742827 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.739595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:44241 (local address 127.0.0.1:39988)
1026 15:07:44.739803 (+ 208us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:44.739833 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:44.739902 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:44.740250 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:44.740253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:44.740255 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:44.740589 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.740591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.741978 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.741979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:44.742592 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.742593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.742660 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.742665 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:07:44.742709 (+ 44us) client_negotiation.cc:240] Negotiation successful
1026 15:07:44.742750 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":62,"threads_started":1}
I20251026 15:07:44.743294 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.739685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.14.212.66:44241)
1026 15:07:44.739858 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:44.739868 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:44.739880 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:07:44.740026 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:44.740029 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:44.740049 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:44.740115 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:44.740736 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.741866 (+ 1130us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.742687 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.743151 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.743196 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:44.743221 (+ 25us) server_negotiation.cc:300] Negotiation successful
1026 15:07:44.743246 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20251026 15:07:44.743475 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.739792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:38959 (local address 127.0.0.1:47104)
1026 15:07:44.740144 (+ 352us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:44.740155 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:44.740214 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:44.740365 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:44.740366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:44.740368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:44.740605 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.740608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.741535 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.741536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:44.742263 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.742265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.742332 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.742366 (+ 34us) client_negotiation.cc:769] Sending connection context
1026 15:07:44.743367 (+ 1001us) client_negotiation.cc:240] Negotiation successful
1026 15:07:44.743408 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":76,"threads_started":1}
I20251026 15:07:44.743744 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.739998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.14.212.65:38959)
1026 15:07:44.740152 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:44.740161 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:44.740200 (+ 39us) server_negotiation.cc:408] Connection header received
1026 15:07:44.740285 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:44.740287 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:44.740299 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:44.740359 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:44.740694 (+ 335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.741443 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.742745 (+ 1302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.743255 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.743305 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:44.743587 (+ 282us) server_negotiation.cc:300] Negotiation successful
1026 15:07:44.743641 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20251026 15:07:44.743775 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.739624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:41061 (local address 127.0.0.1:43336)
1026 15:07:44.739925 (+ 301us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:44.739936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:44.740023 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:44.740294 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:44.740295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:44.740297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:44.740589 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.740591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.741947 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.741947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:44.742831 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.742834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.743632 (+ 798us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.743636 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:07:44.743672 (+ 36us) client_negotiation.cc:240] Negotiation successful
1026 15:07:44.743710 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":84,"threads_started":1}
I20251026 15:07:44.743873 17272 tablet_service.cc:1505] Processing CreateTablet for tablet b8f6607ce55c498e842ab4c7f8c4be0b (DEFAULT_TABLE table=test-table [id=189f12f7ba8340978cf394c430768e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:44.744200 17272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f6607ce55c498e842ab4c7f8c4be0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:44.744292 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.739783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.14.212.67:41061)
1026 15:07:44.739946 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:44.739953 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:44.740082 (+ 129us) server_negotiation.cc:408] Connection header received
1026 15:07:44.740120 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:44.740121 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:44.740138 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:44.740193 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:44.740679 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.741857 (+ 1178us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.743241 (+ 1384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.744127 (+ 886us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.744160 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:44.744184 (+ 24us) server_negotiation.cc:300] Negotiation successful
1026 15:07:44.744212 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20251026 15:07:44.744774 17415 tablet_service.cc:1505] Processing CreateTablet for tablet b8f6607ce55c498e842ab4c7f8c4be0b (DEFAULT_TABLE table=test-table [id=189f12f7ba8340978cf394c430768e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:44.745065 17415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f6607ce55c498e842ab4c7f8c4be0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:44.746780 17129 tablet_service.cc:1505] Processing CreateTablet for tablet b8f6607ce55c498e842ab4c7f8c4be0b (DEFAULT_TABLE table=test-table [id=189f12f7ba8340978cf394c430768e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:44.746914 17504 tablet_bootstrap.cc:492] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c: Bootstrap starting.
I20251026 15:07:44.747056 17129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f6607ce55c498e842ab4c7f8c4be0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:44.747400 17505 tablet_bootstrap.cc:492] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6: Bootstrap starting.
I20251026 15:07:44.747547 17504 tablet_bootstrap.cc:654] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:44.748060 17505 tablet_bootstrap.cc:654] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:44.748237 17504 log.cc:826] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:44.748589 17505 log.cc:826] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:44.748970 17504 tablet_bootstrap.cc:492] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c: No bootstrap required, opened a new log
I20251026 15:07:44.749027 17504 ts_tablet_manager.cc:1403] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:07:44.749248 17505 tablet_bootstrap.cc:492] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6: No bootstrap required, opened a new log
I20251026 15:07:44.749298 17505 ts_tablet_manager.cc:1403] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:07:44.749455 17508 tablet_bootstrap.cc:492] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2: Bootstrap starting.
I20251026 15:07:44.750006 17508 tablet_bootstrap.cc:654] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:44.750483 17508 log.cc:826] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:44.750617 17504 raft_consensus.cc:359] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } }
I20251026 15:07:44.750771 17504 raft_consensus.cc:385] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:44.750805 17504 raft_consensus.cc:740] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf317d62a3f0473fa8e54db7d365221c, State: Initialized, Role: FOLLOWER
I20251026 15:07:44.750818 17505 raft_consensus.cc:359] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } }
I20251026 15:07:44.750934 17505 raft_consensus.cc:385] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:44.750968 17505 raft_consensus.cc:740] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e28fc72acb34dedbad811e3b12702b6, State: Initialized, Role: FOLLOWER
I20251026 15:07:44.750922 17504 consensus_queue.cc:260] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } }
I20251026 15:07:44.751065 17505 consensus_queue.cc:260] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } }
I20251026 15:07:44.751150 17508 tablet_bootstrap.cc:492] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2: No bootstrap required, opened a new log
I20251026 15:07:44.751197 17504 ts_tablet_manager.cc:1434] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:07:44.751209 17508 ts_tablet_manager.cc:1403] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 15:07:44.751282 17505 ts_tablet_manager.cc:1434] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:07:44.751307 17339 heartbeater.cc:499] Master 127.14.212.126:42195 was elected leader, sending a full tablet report...
I20251026 15:07:44.751368 17481 heartbeater.cc:499] Master 127.14.212.126:42195 was elected leader, sending a full tablet report...
I20251026 15:07:44.752605 17508 raft_consensus.cc:359] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } }
I20251026 15:07:44.752712 17508 raft_consensus.cc:385] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:44.752750 17508 raft_consensus.cc:740] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f9f52d8faf498e8ddb1f470bce5bd2, State: Initialized, Role: FOLLOWER
I20251026 15:07:44.752849 17508 consensus_queue.cc:260] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } }
I20251026 15:07:44.753055 17508 ts_tablet_manager.cc:1434] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:07:44.753167 17196 heartbeater.cc:499] Master 127.14.212.126:42195 was elected leader, sending a full tablet report...
W20251026 15:07:44.774652 17340 tablet.cc:2378] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 15:07:44.851496 17197 tablet.cc:2378] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 15:07:44.953562 17482 tablet.cc:2378] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:07:44.990715 17511 raft_consensus.cc:493] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:07:44.990856 17511 raft_consensus.cc:515] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } }
I20251026 15:07:44.991199 17511 leader_election.cc:290] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf317d62a3f0473fa8e54db7d365221c (127.14.212.66:44241), 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65:38959)
I20251026 15:07:44.993343 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.991398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:44241 (local address 127.14.212.67:54463)
1026 15:07:44.991460 (+ 62us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:44.991473 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:44.991547 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:44.991708 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:44.991709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:44.991712 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:44.991881 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.991883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.992563 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.992563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:44.993128 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.993129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.993199 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.993204 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:07:44.993237 (+ 33us) client_negotiation.cc:240] Negotiation successful
1026 15:07:44.993274 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20251026 15:07:44.993716 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.991509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:38959 (local address 127.14.212.67:57249)
1026 15:07:44.991674 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:44.991684 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:44.991746 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:44.992018 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:44.992019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:44.992022 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:44.992222 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.992223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.992873 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.992874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:44.993445 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:44.993447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.993556 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.993560 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:07:44.993591 (+ 31us) client_negotiation.cc:240] Negotiation successful
1026 15:07:44.993623 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
I20251026 15:07:44.993791 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.991508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:54463 (local address 127.14.212.66:44241)
1026 15:07:44.991561 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:44.991563 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:44.991576 (+ 13us) server_negotiation.cc:408] Connection header received
1026 15:07:44.991609 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:44.991610 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:44.991621 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:44.991676 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:44.991985 (+ 309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.992475 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.993214 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.993670 (+ 456us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.993678 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:44.993699 (+ 21us) server_negotiation.cc:300] Negotiation successful
1026 15:07:44.993731 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251026 15:07:44.994153 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:44.991587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:57249 (local address 127.14.212.65:38959)
1026 15:07:44.991642 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:44.991644 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:44.991743 (+ 99us) server_negotiation.cc:408] Connection header received
1026 15:07:44.991806 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:44.991807 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:44.991818 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:44.991988 (+ 170us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:44.992302 (+ 314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.992787 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:44.993590 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:44.994022 (+ 432us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:44.994029 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:44.994057 (+ 28us) server_negotiation.cc:300] Negotiation successful
1026 15:07:44.994088 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251026 15:07:44.994122 17292 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8f6607ce55c498e842ab4c7f8c4be0b" candidate_uuid: "0e28fc72acb34dedbad811e3b12702b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf317d62a3f0473fa8e54db7d365221c" is_pre_election: true
I20251026 15:07:44.994273 17292 raft_consensus.cc:2468] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e28fc72acb34dedbad811e3b12702b6 in term 0.
I20251026 15:07:44.994480 17145 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8f6607ce55c498e842ab4c7f8c4be0b" candidate_uuid: "0e28fc72acb34dedbad811e3b12702b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" is_pre_election: true
I20251026 15:07:44.994510 17368 leader_election.cc:304] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e28fc72acb34dedbad811e3b12702b6, bf317d62a3f0473fa8e54db7d365221c; no voters:
I20251026 15:07:44.994611 17145 raft_consensus.cc:2468] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e28fc72acb34dedbad811e3b12702b6 in term 0.
I20251026 15:07:44.994621 17511 raft_consensus.cc:2804] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:07:44.994674 17511 raft_consensus.cc:493] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:07:44.994704 17511 raft_consensus.cc:3060] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:44.995391 17511 raft_consensus.cc:515] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } }
I20251026 15:07:44.995504 17511 leader_election.cc:290] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [CANDIDATE]: Term 1 election: Requested vote from peers bf317d62a3f0473fa8e54db7d365221c (127.14.212.66:44241), 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65:38959)
I20251026 15:07:44.995671 17292 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8f6607ce55c498e842ab4c7f8c4be0b" candidate_uuid: "0e28fc72acb34dedbad811e3b12702b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf317d62a3f0473fa8e54db7d365221c"
I20251026 15:07:44.995697 17145 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8f6607ce55c498e842ab4c7f8c4be0b" candidate_uuid: "0e28fc72acb34dedbad811e3b12702b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2"
I20251026 15:07:44.995730 17292 raft_consensus.cc:3060] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:44.995761 17145 raft_consensus.cc:3060] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:44.996392 17145 raft_consensus.cc:2468] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e28fc72acb34dedbad811e3b12702b6 in term 1.
I20251026 15:07:44.996397 17292 raft_consensus.cc:2468] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e28fc72acb34dedbad811e3b12702b6 in term 1.
I20251026 15:07:44.996562 17368 leader_election.cc:304] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e28fc72acb34dedbad811e3b12702b6, 33f9f52d8faf498e8ddb1f470bce5bd2; no voters:
I20251026 15:07:44.996752 17511 raft_consensus.cc:2804] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:44.996963 17511 raft_consensus.cc:697] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [term 1 LEADER]: Becoming Leader. State: Replica: 0e28fc72acb34dedbad811e3b12702b6, State: Running, Role: LEADER
I20251026 15:07:44.997076 17511 consensus_queue.cc:237] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } }
I20251026 15:07:44.998013 16988 catalog_manager.cc:5649] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e28fc72acb34dedbad811e3b12702b6 (127.14.212.67). New cstate: current_term: 1 leader_uuid: "0e28fc72acb34dedbad811e3b12702b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } health_report { overall_health: UNKNOWN } } }
Oct 26 15:07:45 dist-test-slave-8pgz krb5kdc[16941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491264, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:45.049357 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.043304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:42195 (local address 127.0.0.1:35160)
1026 15:07:45.043453 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:45.043463 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:45.043528 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:45.043659 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:45.043660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:45.044046 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:45.044211 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.044214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.044843 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.044844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:45.045220 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.045222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.045286 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.045837 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:45.045841 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:45.047249 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:45.048607 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:45.048613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:45.048616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:45.048819 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:45.048936 (+ 117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:45.048936 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:45.048937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:45.048963 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:45.049112 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:45.049115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:45.049150 (+ 35us) client_negotiation.cc:769] Sending connection context
1026 15:07:45.049200 (+ 50us) client_negotiation.cc:240] Negotiation successful
1026 15:07:45.049286 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20251026 15:07:45.049429 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.043386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35160 (local address 127.14.212.126:42195)
1026 15:07:45.043475 (+ 89us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:45.043477 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:45.043527 (+ 50us) server_negotiation.cc:408] Connection header received
1026 15:07:45.043573 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:45.043574 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:45.043583 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:45.043649 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:45.044279 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.044753 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.045312 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.045434 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.047324 (+ 1890us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:45.047329 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:45.047330 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:45.047336 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:45.048542 (+ 1206us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:45.048875 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:45.048876 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:45.048876 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:45.048892 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:45.049002 (+ 110us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:45.049003 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:45.049003 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:45.049073 (+ 70us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:45.049117 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:45.049218 (+ 101us) server_negotiation.cc:300] Negotiation successful
1026 15:07:45.049336 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":69}
I20251026 15:07:45.051312 16988 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:35160:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251026 15:07:45.051517 16988 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:07:45.052901 17129 tablet_service.cc:1505] Processing CreateTablet for tablet b283e15f7b3a46b09edfb36f03240e30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff603da4094445ada7b57ce418fe9e14]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 15:07:45.052979 17415 tablet_service.cc:1505] Processing CreateTablet for tablet b283e15f7b3a46b09edfb36f03240e30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff603da4094445ada7b57ce418fe9e14]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 15:07:45.052958 17272 tablet_service.cc:1505] Processing CreateTablet for tablet b283e15f7b3a46b09edfb36f03240e30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff603da4094445ada7b57ce418fe9e14]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 15:07:45.053064 17129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b283e15f7b3a46b09edfb36f03240e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:45.053112 17272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b283e15f7b3a46b09edfb36f03240e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:45.053112 17415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b283e15f7b3a46b09edfb36f03240e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:45.054141 17508 tablet_bootstrap.cc:492] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2: Bootstrap starting.
I20251026 15:07:45.054566 17508 tablet_bootstrap.cc:654] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:45.054553 17504 tablet_bootstrap.cc:492] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c: Bootstrap starting.
I20251026 15:07:45.054986 17504 tablet_bootstrap.cc:654] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:45.055246 17508 tablet_bootstrap.cc:492] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2: No bootstrap required, opened a new log
I20251026 15:07:45.055284 17505 tablet_bootstrap.cc:492] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6: Bootstrap starting.
I20251026 15:07:45.055285 17508 ts_tablet_manager.cc:1403] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:45.055428 17508 raft_consensus.cc:359] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.055500 17508 raft_consensus.cc:385] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:45.055524 17508 raft_consensus.cc:740] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f9f52d8faf498e8ddb1f470bce5bd2, State: Initialized, Role: FOLLOWER
I20251026 15:07:45.055527 17504 tablet_bootstrap.cc:492] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c: No bootstrap required, opened a new log
I20251026 15:07:45.055562 17504 ts_tablet_manager.cc:1403] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:45.055575 17508 consensus_queue.cc:260] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.055672 17508 ts_tablet_manager.cc:1434] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:45.055696 17504 raft_consensus.cc:359] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.055721 17505 tablet_bootstrap.cc:654] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:45.055759 17504 raft_consensus.cc:385] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:45.055779 17504 raft_consensus.cc:740] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf317d62a3f0473fa8e54db7d365221c, State: Initialized, Role: FOLLOWER
I20251026 15:07:45.055824 17504 consensus_queue.cc:260] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.055920 17504 ts_tablet_manager.cc:1434] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:45.055992 17510 tablet_replica.cc:442] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } }
I20251026 15:07:45.056074 17510 tablet_replica.cc:445] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:45.056258 17512 tablet_replica.cc:442] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } }
I20251026 15:07:45.056329 17512 tablet_replica.cc:445] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:45.056783 17505 tablet_bootstrap.cc:492] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6: No bootstrap required, opened a new log
I20251026 15:07:45.056828 17505 ts_tablet_manager.cc:1403] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:07:45.056953 17505 raft_consensus.cc:359] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.057008 17505 raft_consensus.cc:385] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:45.057021 17505 raft_consensus.cc:740] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e28fc72acb34dedbad811e3b12702b6, State: Initialized, Role: FOLLOWER
I20251026 15:07:45.057055 17505 consensus_queue.cc:260] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.057163 17505 ts_tablet_manager.cc:1434] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:45.057209 17511 tablet_replica.cc:442] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } }
I20251026 15:07:45.057277 17511 tablet_replica.cc:445] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:45.104329 17511 raft_consensus.cc:493] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:07:45.104431 17511 raft_consensus.cc:515] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.104581 17511 leader_election.cc:290] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65:38959), bf317d62a3f0473fa8e54db7d365221c (127.14.212.66:44241)
I20251026 15:07:45.104810 17292 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b283e15f7b3a46b09edfb36f03240e30" candidate_uuid: "0e28fc72acb34dedbad811e3b12702b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf317d62a3f0473fa8e54db7d365221c" is_pre_election: true
I20251026 15:07:45.104848 17145 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b283e15f7b3a46b09edfb36f03240e30" candidate_uuid: "0e28fc72acb34dedbad811e3b12702b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" is_pre_election: true
I20251026 15:07:45.104902 17292 raft_consensus.cc:2468] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e28fc72acb34dedbad811e3b12702b6 in term 0.
I20251026 15:07:45.104939 17145 raft_consensus.cc:2468] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e28fc72acb34dedbad811e3b12702b6 in term 0.
I20251026 15:07:45.105118 17368 leader_election.cc:304] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e28fc72acb34dedbad811e3b12702b6, 33f9f52d8faf498e8ddb1f470bce5bd2; no voters:
I20251026 15:07:45.105229 17511 raft_consensus.cc:2804] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:07:45.105284 17511 raft_consensus.cc:493] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:07:45.105301 17511 raft_consensus.cc:3060] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:45.106006 17511 raft_consensus.cc:515] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.106125 17511 leader_election.cc:290] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [CANDIDATE]: Term 1 election: Requested vote from peers 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65:38959), bf317d62a3f0473fa8e54db7d365221c (127.14.212.66:44241)
I20251026 15:07:45.106297 17292 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b283e15f7b3a46b09edfb36f03240e30" candidate_uuid: "0e28fc72acb34dedbad811e3b12702b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf317d62a3f0473fa8e54db7d365221c"
I20251026 15:07:45.106295 17145 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b283e15f7b3a46b09edfb36f03240e30" candidate_uuid: "0e28fc72acb34dedbad811e3b12702b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2"
I20251026 15:07:45.106360 17145 raft_consensus.cc:3060] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:45.106360 17292 raft_consensus.cc:3060] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:45.106981 17145 raft_consensus.cc:2468] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e28fc72acb34dedbad811e3b12702b6 in term 1.
I20251026 15:07:45.107035 17292 raft_consensus.cc:2468] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e28fc72acb34dedbad811e3b12702b6 in term 1.
I20251026 15:07:45.107170 17368 leader_election.cc:304] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e28fc72acb34dedbad811e3b12702b6, 33f9f52d8faf498e8ddb1f470bce5bd2; no voters:
I20251026 15:07:45.107271 17511 raft_consensus.cc:2804] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:45.107358 17511 raft_consensus.cc:697] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [term 1 LEADER]: Becoming Leader. State: Replica: 0e28fc72acb34dedbad811e3b12702b6, State: Running, Role: LEADER
I20251026 15:07:45.107420 17511 consensus_queue.cc:237] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.107725 17514 tablet_replica.cc:442] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6: TxnStatusTablet state changed. Reason: New leader 0e28fc72acb34dedbad811e3b12702b6. Latest consensus state: current_term: 1 leader_uuid: "0e28fc72acb34dedbad811e3b12702b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } }
I20251026 15:07:45.107789 17514 tablet_replica.cc:445] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6: This TxnStatusTablet replica's current role is: LEADER
I20251026 15:07:45.107961 17526 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251026 15:07:45.108073 17526 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251026 15:07:45.108022 16988 catalog_manager.cc:5649] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e28fc72acb34dedbad811e3b12702b6 (127.14.212.67). New cstate: current_term: 1 leader_uuid: "0e28fc72acb34dedbad811e3b12702b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } health_report { overall_health: HEALTHY } } }
I20251026 15:07:45.115832 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.113565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.14.212.67:41061)
1026 15:07:45.113626 (+ 61us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:45.113627 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:45.113635 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:07:45.113723 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:45.113723 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:45.113737 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:45.113787 (+ 50us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:45.114098 (+ 311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.114596 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.115383 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.115498 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.115521 (+ 23us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:45.115608 (+ 87us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:45.115680 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:45.115768 (+ 88us) server_negotiation.cc:300] Negotiation successful
1026 15:07:45.115787 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20251026 15:07:45.115886 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.113510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:41061 (local address 127.0.0.1:43350)
1026 15:07:45.113596 (+ 86us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:45.113607 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:45.113653 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:45.113810 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:45.113811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:45.113812 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 15:07:45.114034 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.114036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.114669 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.114670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:45.115310 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.115312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.115389 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.115406 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 15:07:45.115702 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 15:07:45.115709 (+ 7us) client_negotiation.cc:769] Sending connection context
1026 15:07:45.115763 (+ 54us) client_negotiation.cc:240] Negotiation successful
1026 15:07:45.115812 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20251026 15:07:45.117488 17292 raft_consensus.cc:1275] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Refusing update from remote peer 0e28fc72acb34dedbad811e3b12702b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:45.117556 17145 raft_consensus.cc:1275] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Refusing update from remote peer 0e28fc72acb34dedbad811e3b12702b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:45.117753 17511 consensus_queue.cc:1048] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:45.117854 17514 consensus_queue.cc:1048] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:45.118882 17510 tablet_replica.cc:442] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c: TxnStatusTablet state changed. Reason: New leader 0e28fc72acb34dedbad811e3b12702b6. Latest consensus state: current_term: 1 leader_uuid: "0e28fc72acb34dedbad811e3b12702b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } }
I20251026 15:07:45.118966 17510 tablet_replica.cc:445] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:45.119038 17512 tablet_replica.cc:442] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2: TxnStatusTablet state changed. Reason: New leader 0e28fc72acb34dedbad811e3b12702b6. Latest consensus state: current_term: 1 leader_uuid: "0e28fc72acb34dedbad811e3b12702b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } }
I20251026 15:07:45.119102 17512 tablet_replica.cc:445] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:45.119809 17511 tablet_replica.cc:442] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e28fc72acb34dedbad811e3b12702b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } }
I20251026 15:07:45.119903 17511 tablet_replica.cc:445] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6: This TxnStatusTablet replica's current role is: LEADER
I20251026 15:07:45.120078 17529 mvcc.cc:204] Tried to move back new op lower bound from 7215068221918945280 to 7215068221880258560. Current Snapshot: MvccSnapshot[applied={T|T < 7215068221918945280}]
I20251026 15:07:45.120540 17510 tablet_replica.cc:442] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e28fc72acb34dedbad811e3b12702b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } }
I20251026 15:07:45.120613 17510 tablet_replica.cc:445] T b283e15f7b3a46b09edfb36f03240e30 P bf317d62a3f0473fa8e54db7d365221c: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:45.120723 17530 mvcc.cc:204] Tried to move back new op lower bound from 7215068221918945280 to 7215068221880258560. Current Snapshot: MvccSnapshot[applied={T|T < 7215068221918945280}]
I20251026 15:07:45.121897 17514 tablet_replica.cc:442] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e28fc72acb34dedbad811e3b12702b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } }
I20251026 15:07:45.121984 17514 tablet_replica.cc:445] T b283e15f7b3a46b09edfb36f03240e30 P 0e28fc72acb34dedbad811e3b12702b6: This TxnStatusTablet replica's current role is: LEADER
I20251026 15:07:45.122627 17512 tablet_replica.cc:442] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e28fc72acb34dedbad811e3b12702b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } } }
I20251026 15:07:45.122720 17512 tablet_replica.cc:445] T b283e15f7b3a46b09edfb36f03240e30 P 33f9f52d8faf498e8ddb1f470bce5bd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:07:45.124197 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.121807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.14.212.67:41061)
1026 15:07:45.121843 (+ 36us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:45.121844 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:45.121852 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:07:45.121873 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:45.121874 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:45.121884 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:45.121938 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:45.122409 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.123044 (+ 635us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.123694 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.123862 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.123883 (+ 21us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:45.123910 (+ 27us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:45.123987 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:45.124110 (+ 123us) server_negotiation.cc:300] Negotiation successful
1026 15:07:45.124134 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20251026 15:07:45.128512 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.126301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:41061 (local address 127.14.212.67:34167)
1026 15:07:45.126367 (+ 66us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:45.126396 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:45.126451 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:45.126595 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:45.126596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:45.126597 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 15:07:45.126758 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.126759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.127620 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.127620 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:45.128027 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.128029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.128082 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.128094 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 15:07:45.128363 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 15:07:45.128364 (+ 1us) client_negotiation.cc:769] Sending connection context
1026 15:07:45.128395 (+ 31us) client_negotiation.cc:240] Negotiation successful
1026 15:07:45.128423 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20251026 15:07:45.128520 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.126362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:34167 (local address 127.14.212.67:41061)
1026 15:07:45.126446 (+ 84us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:45.126448 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:45.126457 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:07:45.126513 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:45.126514 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:45.126522 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:45.126593 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:45.126815 (+ 222us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.127555 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.128092 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.128256 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.128275 (+ 19us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:45.128299 (+ 24us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:45.128351 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:45.128410 (+ 59us) server_negotiation.cc:300] Negotiation successful
1026 15:07:45.128430 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251026 15:07:45.130502 17292 raft_consensus.cc:1275] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Refusing update from remote peer 0e28fc72acb34dedbad811e3b12702b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:45.130564 17145 raft_consensus.cc:1275] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Refusing update from remote peer 0e28fc72acb34dedbad811e3b12702b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:45.130671 17540 consensus_queue.cc:1048] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:45.130769 17540 consensus_queue.cc:1048] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:45.132324 17531 mvcc.cc:204] Tried to move back new op lower bound from 7215068221973295104 to 7215068221428740096. Current Snapshot: MvccSnapshot[applied={T|T < 7215068221973295104}]
I20251026 15:07:45.142520 16988 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:07:45.142648 16988 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:07:45.143393 16988 catalog_manager.cc:5936] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606: Sending DeleteTablet for 3 replicas of tablet b8f6607ce55c498e842ab4c7f8c4be0b
I20251026 15:07:45.143700 17272 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f6607ce55c498e842ab4c7f8c4be0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:45 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:39988
I20251026 15:07:45.143779 17415 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f6607ce55c498e842ab4c7f8c4be0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:45 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:43336
I20251026 15:07:45.143923 17129 tablet_service.cc:1552] Processing DeleteTablet for tablet b8f6607ce55c498e842ab4c7f8c4be0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:45 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:47104
I20251026 15:07:45.144068 17544 tablet_replica.cc:333] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c: stopping tablet replica
I20251026 15:07:45.144167 17544 raft_consensus.cc:2243] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:07:45.144229 17544 raft_consensus.cc:2272] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:45.144186 17546 tablet_replica.cc:333] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2: stopping tablet replica
I20251026 15:07:45.144364 16988 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:07:45.144508 16988 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:07:45.144448 17546 raft_consensus.cc:2243] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:07:45.144675 17544 ts_tablet_manager.cc:1916] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:45.144802 17546 raft_consensus.cc:2272] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:45.145241 17546 ts_tablet_manager.cc:1916] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:45.145946 17415 tablet_service.cc:1505] Processing CreateTablet for tablet 9e80ef4972894135a198ef566a4f26cc (DEFAULT_TABLE table=test-table [id=c8bce1704ea04c7a8088b0e7f09ac967]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:45.146068 17415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e80ef4972894135a198ef566a4f26cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:45.146386 17544 ts_tablet_manager.cc:1929] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:07:45.146485 17544 log.cc:1199] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/b8f6607ce55c498e842ab4c7f8c4be0b
I20251026 15:07:45.146545 17272 tablet_service.cc:1505] Processing CreateTablet for tablet 9e80ef4972894135a198ef566a4f26cc (DEFAULT_TABLE table=test-table [id=c8bce1704ea04c7a8088b0e7f09ac967]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:45.146664 17272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e80ef4972894135a198ef566a4f26cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:45.146759 17544 ts_tablet_manager.cc:1950] T b8f6607ce55c498e842ab4c7f8c4be0b P bf317d62a3f0473fa8e54db7d365221c: Deleting consensus metadata
I20251026 15:07:45.147125 17545 tablet_replica.cc:333] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6: stopping tablet replica
I20251026 15:07:45.147217 17545 raft_consensus.cc:2243] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:07:45.147321 17545 raft_consensus.cc:2272] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:45.147603 17546 ts_tablet_manager.cc:1929] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:07:45.147713 17546 log.cc:1199] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/wal/wals/b8f6607ce55c498e842ab4c7f8c4be0b
I20251026 15:07:45.147754 17545 ts_tablet_manager.cc:1916] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:45.147820 17504 tablet_bootstrap.cc:492] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c: Bootstrap starting.
I20251026 15:07:45.148085 17546 ts_tablet_manager.cc:1950] T b8f6607ce55c498e842ab4c7f8c4be0b P 33f9f52d8faf498e8ddb1f470bce5bd2: Deleting consensus metadata
I20251026 15:07:45.148397 17504 tablet_bootstrap.cc:654] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:45.148793 16975 catalog_manager.cc:4985] TS bf317d62a3f0473fa8e54db7d365221c (127.14.212.66:44241): tablet b8f6607ce55c498e842ab4c7f8c4be0b (table test-table [id=189f12f7ba8340978cf394c430768e32]) successfully deleted
I20251026 15:07:45.149296 17545 ts_tablet_manager.cc:1929] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:07:45.149358 17545 log.cc:1199] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/b8f6607ce55c498e842ab4c7f8c4be0b
I20251026 15:07:45.149578 17545 ts_tablet_manager.cc:1950] T b8f6607ce55c498e842ab4c7f8c4be0b P 0e28fc72acb34dedbad811e3b12702b6: Deleting consensus metadata
I20251026 15:07:45.149879 16974 catalog_manager.cc:4985] TS 0e28fc72acb34dedbad811e3b12702b6 (127.14.212.67:41061): tablet b8f6607ce55c498e842ab4c7f8c4be0b (table test-table [id=189f12f7ba8340978cf394c430768e32]) successfully deleted
I20251026 15:07:45.149959 17505 tablet_bootstrap.cc:492] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6: Bootstrap starting.
I20251026 15:07:45.150233 17504 tablet_bootstrap.cc:492] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c: No bootstrap required, opened a new log
I20251026 15:07:45.150274 17504 ts_tablet_manager.cc:1403] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:07:45.150313 17505 tablet_bootstrap.cc:654] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:45.150434 17504 raft_consensus.cc:359] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.150494 17504 raft_consensus.cc:385] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:45.150521 17504 raft_consensus.cc:740] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf317d62a3f0473fa8e54db7d365221c, State: Initialized, Role: FOLLOWER
I20251026 15:07:45.150560 17504 consensus_queue.cc:260] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.150646 17504 ts_tablet_manager.cc:1434] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:45.150904 17505 tablet_bootstrap.cc:492] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6: No bootstrap required, opened a new log
I20251026 15:07:45.150954 17505 ts_tablet_manager.cc:1403] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 15:07:45.151011 17129 tablet_service.cc:1505] Processing CreateTablet for tablet 9e80ef4972894135a198ef566a4f26cc (DEFAULT_TABLE table=test-table [id=c8bce1704ea04c7a8088b0e7f09ac967]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:45.151103 17129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e80ef4972894135a198ef566a4f26cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:45.151104 17505 raft_consensus.cc:359] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.151146 17505 raft_consensus.cc:385] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:45.151160 17505 raft_consensus.cc:740] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e28fc72acb34dedbad811e3b12702b6, State: Initialized, Role: FOLLOWER
I20251026 15:07:45.151196 17505 consensus_queue.cc:260] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.151293 17505 ts_tablet_manager.cc:1434] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:45.152290 17508 tablet_bootstrap.cc:492] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2: Bootstrap starting.
I20251026 15:07:45.152515 16975 catalog_manager.cc:4985] TS 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65:38959): tablet b8f6607ce55c498e842ab4c7f8c4be0b (table test-table [id=189f12f7ba8340978cf394c430768e32]) successfully deleted
I20251026 15:07:45.152851 17508 tablet_bootstrap.cc:654] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:45.153414 17508 tablet_bootstrap.cc:492] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2: No bootstrap required, opened a new log
I20251026 15:07:45.153458 17508 ts_tablet_manager.cc:1403] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:45.153609 17508 raft_consensus.cc:359] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.153656 17508 raft_consensus.cc:385] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:45.153679 17508 raft_consensus.cc:740] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f9f52d8faf498e8ddb1f470bce5bd2, State: Initialized, Role: FOLLOWER
I20251026 15:07:45.153760 17508 consensus_queue.cc:260] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.153853 17508 ts_tablet_manager.cc:1434] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:45.249209 17510 raft_consensus.cc:493] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:07:45.249320 17510 raft_consensus.cc:515] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.249629 17510 leader_election.cc:290] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65:38959), 0e28fc72acb34dedbad811e3b12702b6 (127.14.212.67:41061)
I20251026 15:07:45.252332 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.249820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:38959 (local address 127.14.212.66:40947)
1026 15:07:45.250029 (+ 209us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:45.250041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:45.250123 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:45.250349 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:45.250350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:45.250351 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:45.250631 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.250634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.251481 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.251483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:45.252104 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.252106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.252174 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.252179 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:07:45.252212 (+ 33us) client_negotiation.cc:240] Negotiation successful
1026 15:07:45.252247 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":59,"threads_started":1}
I20251026 15:07:45.252746 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.249975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:40947 (local address 127.14.212.65:38959)
1026 15:07:45.250021 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:45.250023 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:45.250121 (+ 98us) server_negotiation.cc:408] Connection header received
1026 15:07:45.250221 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:45.250223 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:45.250236 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:45.250268 (+ 32us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:45.250717 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.251419 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.252187 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.252604 (+ 417us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.252614 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:45.252638 (+ 24us) server_negotiation.cc:300] Negotiation successful
1026 15:07:45.252675 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251026 15:07:45.252936 17145 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e80ef4972894135a198ef566a4f26cc" candidate_uuid: "bf317d62a3f0473fa8e54db7d365221c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" is_pre_election: true
I20251026 15:07:45.253021 17145 raft_consensus.cc:2468] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf317d62a3f0473fa8e54db7d365221c in term 0.
I20251026 15:07:45.253150 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.249821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:41061 (local address 127.14.212.66:36665)
1026 15:07:45.250081 (+ 260us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:45.250089 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:45.250158 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:45.250305 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:45.250306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:45.250308 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:45.250549 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.250551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.251445 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.251446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:45.252357 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.252360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.252958 (+ 598us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.252962 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:07:45.253023 (+ 61us) client_negotiation.cc:240] Negotiation successful
1026 15:07:45.253064 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":10,"thread_start_us":80,"threads_started":1}
I20251026 15:07:45.253188 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.249929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:36665 (local address 127.14.212.67:41061)
1026 15:07:45.249981 (+ 52us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:45.249983 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:45.250148 (+ 165us) server_negotiation.cc:408] Connection header received
1026 15:07:45.250232 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:45.250233 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:45.250241 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:45.250297 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:45.250624 (+ 327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.251343 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.252460 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.252910 (+ 450us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.252917 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:45.253073 (+ 156us) server_negotiation.cc:300] Negotiation successful
1026 15:07:45.253127 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251026 15:07:45.253208 17225 leader_election.cc:304] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33f9f52d8faf498e8ddb1f470bce5bd2, bf317d62a3f0473fa8e54db7d365221c; no voters:
I20251026 15:07:45.253415 17510 raft_consensus.cc:2804] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:07:45.253451 17510 raft_consensus.cc:493] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:07:45.253471 17510 raft_consensus.cc:3060] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:45.253535 17435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e80ef4972894135a198ef566a4f26cc" candidate_uuid: "bf317d62a3f0473fa8e54db7d365221c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e28fc72acb34dedbad811e3b12702b6" is_pre_election: true
I20251026 15:07:45.253652 17435 raft_consensus.cc:2468] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf317d62a3f0473fa8e54db7d365221c in term 0.
I20251026 15:07:45.254175 17510 raft_consensus.cc:515] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.254312 17510 leader_election.cc:290] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [CANDIDATE]: Term 1 election: Requested vote from peers 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65:38959), 0e28fc72acb34dedbad811e3b12702b6 (127.14.212.67:41061)
I20251026 15:07:45.254484 17145 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e80ef4972894135a198ef566a4f26cc" candidate_uuid: "bf317d62a3f0473fa8e54db7d365221c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2"
I20251026 15:07:45.254516 17435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e80ef4972894135a198ef566a4f26cc" candidate_uuid: "bf317d62a3f0473fa8e54db7d365221c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e28fc72acb34dedbad811e3b12702b6"
I20251026 15:07:45.254555 17145 raft_consensus.cc:3060] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:45.254580 17435 raft_consensus.cc:3060] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:45.255146 17145 raft_consensus.cc:2468] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf317d62a3f0473fa8e54db7d365221c in term 1.
I20251026 15:07:45.255282 17225 leader_election.cc:304] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33f9f52d8faf498e8ddb1f470bce5bd2, bf317d62a3f0473fa8e54db7d365221c; no voters:
I20251026 15:07:45.255395 17510 raft_consensus.cc:2804] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:45.255498 17435 raft_consensus.cc:2468] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf317d62a3f0473fa8e54db7d365221c in term 1.
I20251026 15:07:45.255530 17510 raft_consensus.cc:697] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [term 1 LEADER]: Becoming Leader. State: Replica: bf317d62a3f0473fa8e54db7d365221c, State: Running, Role: LEADER
I20251026 15:07:45.255597 17510 consensus_queue.cc:237] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:45.256250 16988 catalog_manager.cc:5649] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c reported cstate change: term changed from 0 to 1, leader changed from <none> to bf317d62a3f0473fa8e54db7d365221c (127.14.212.66). New cstate: current_term: 1 leader_uuid: "bf317d62a3f0473fa8e54db7d365221c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } health_report { overall_health: UNKNOWN } } }
I20251026 15:07:45.272719 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.270438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.14.212.66:44241)
1026 15:07:45.270493 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:45.270495 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:45.270503 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:07:45.270537 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:45.270538 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:45.270553 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:45.270612 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:45.270934 (+ 322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.271388 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.272257 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.272375 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.272407 (+ 32us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:45.272496 (+ 89us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:45.272557 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:45.272626 (+ 69us) server_negotiation.cc:300] Negotiation successful
1026 15:07:45.272646 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251026 15:07:45.277370 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.275036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:41061 (local address 127.14.212.66:57197)
1026 15:07:45.275200 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:45.275210 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:45.275264 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:45.275404 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:45.275405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:45.275407 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 15:07:45.275652 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.275653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.276213 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.276214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:45.276906 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.276908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.276980 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.276995 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 15:07:45.277189 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 15:07:45.277191 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:07:45.277224 (+ 33us) client_negotiation.cc:240] Negotiation successful
1026 15:07:45.277264 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20251026 15:07:45.277458 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.275111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:57197 (local address 127.14.212.67:41061)
1026 15:07:45.275159 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:45.275161 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:45.275263 (+ 102us) server_negotiation.cc:408] Connection header received
1026 15:07:45.275320 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:45.275321 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:45.275333 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:45.275387 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:45.275712 (+ 325us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.276152 (+ 440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.276977 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.277090 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.277106 (+ 16us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:45.277129 (+ 23us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:45.277173 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:45.277343 (+ 170us) server_negotiation.cc:300] Negotiation successful
1026 15:07:45.277366 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251026 15:07:45.279699 17145 raft_consensus.cc:1275] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Refusing update from remote peer bf317d62a3f0473fa8e54db7d365221c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:45.279824 17435 raft_consensus.cc:1275] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Refusing update from remote peer bf317d62a3f0473fa8e54db7d365221c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:45.279945 17552 consensus_queue.cc:1048] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [LEADER]: Connected to new peer: Peer: permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:45.280043 17510 consensus_queue.cc:1048] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:45.286816 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.284584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:44241 (local address 127.14.212.67:42985)
1026 15:07:45.284623 (+ 39us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:45.284635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:45.284673 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:45.284862 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:45.284863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:45.284865 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 15:07:45.285061 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.285063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.285645 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.285645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:45.286231 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:45.286233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.286302 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.286306 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 15:07:45.286684 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 15:07:45.286686 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:07:45.286727 (+ 41us) client_negotiation.cc:240] Negotiation successful
1026 15:07:45.286768 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20251026 15:07:45.286829 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:45.284682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:42985 (local address 127.14.212.66:44241)
1026 15:07:45.284719 (+ 37us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:45.284721 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:45.284728 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:07:45.284751 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:45.284751 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:45.284762 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:45.284814 (+ 52us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:45.285141 (+ 327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.285575 (+ 434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:45.286370 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:45.286576 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:45.286597 (+ 21us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:45.286622 (+ 25us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:45.286680 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:45.286742 (+ 62us) server_negotiation.cc:300] Negotiation successful
1026 15:07:45.286764 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20251026 15:07:45.287163 17530 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251026 15:07:46.811858 16989 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:07:46.811969 16989 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:07:46.812966 16989 catalog_manager.cc:5936] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606: Sending DeleteTablet for 3 replicas of tablet 9e80ef4972894135a198ef566a4f26cc
I20251026 15:07:46.813231 17129 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e80ef4972894135a198ef566a4f26cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:46 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:47104
I20251026 15:07:46.813326 17272 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e80ef4972894135a198ef566a4f26cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:46 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:39988
I20251026 15:07:46.813446 17584 tablet_replica.cc:333] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2: stopping tablet replica
I20251026 15:07:46.813499 17584 raft_consensus.cc:2243] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:07:46.813555 17584 raft_consensus.cc:2272] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:46.813534 17585 tablet_replica.cc:333] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c: stopping tablet replica
I20251026 15:07:46.813596 17585 raft_consensus.cc:2243] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:07:46.813735 17585 raft_consensus.cc:2272] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:46.813858 17584 ts_tablet_manager.cc:1916] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:46.813989 17585 ts_tablet_manager.cc:1916] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:46.815383 17415 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e80ef4972894135a198ef566a4f26cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:46 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:43336
I20251026 15:07:46.815469 17584 ts_tablet_manager.cc:1929] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251026 15:07:46.815531 17584 log.cc:1199] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/wal/wals/9e80ef4972894135a198ef566a4f26cc
I20251026 15:07:46.815598 17585 ts_tablet_manager.cc:1929] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251026 15:07:46.815667 17585 log.cc:1199] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/9e80ef4972894135a198ef566a4f26cc
I20251026 15:07:46.815830 17584 ts_tablet_manager.cc:1950] T 9e80ef4972894135a198ef566a4f26cc P 33f9f52d8faf498e8ddb1f470bce5bd2: Deleting consensus metadata
I20251026 15:07:46.815948 17585 ts_tablet_manager.cc:1950] T 9e80ef4972894135a198ef566a4f26cc P bf317d62a3f0473fa8e54db7d365221c: Deleting consensus metadata
I20251026 15:07:46.816254 16975 catalog_manager.cc:4985] TS 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65:38959): tablet 9e80ef4972894135a198ef566a4f26cc (table test-table [id=c8bce1704ea04c7a8088b0e7f09ac967]) successfully deleted
I20251026 15:07:46.816341 16975 catalog_manager.cc:4985] TS bf317d62a3f0473fa8e54db7d365221c (127.14.212.66:44241): tablet 9e80ef4972894135a198ef566a4f26cc (table test-table [id=c8bce1704ea04c7a8088b0e7f09ac967]) successfully deleted
I20251026 15:07:46.817857 17587 tablet_replica.cc:333] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6: stopping tablet replica
I20251026 15:07:46.817914 17587 raft_consensus.cc:2243] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:07:46.817951 17587 raft_consensus.cc:2272] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:46.819038 17587 ts_tablet_manager.cc:1916] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:46.820286 17587 ts_tablet_manager.cc:1929] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251026 15:07:46.820341 17587 log.cc:1199] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/9e80ef4972894135a198ef566a4f26cc
I20251026 15:07:46.820581 17587 ts_tablet_manager.cc:1950] T 9e80ef4972894135a198ef566a4f26cc P 0e28fc72acb34dedbad811e3b12702b6: Deleting consensus metadata
I20251026 15:07:46.820931 16974 catalog_manager.cc:4985] TS 0e28fc72acb34dedbad811e3b12702b6 (127.14.212.67:41061): tablet 9e80ef4972894135a198ef566a4f26cc (table test-table [id=c8bce1704ea04c7a8088b0e7f09ac967]) successfully deleted
Oct 26 15:07:46 dist-test-slave-8pgz krb5kdc[16941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491266, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 26 15:07:46 dist-test-slave-8pgz krb5kdc[16941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491266, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:46.836839 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:46.829585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35162 (local address 127.14.212.126:42195)
1026 15:07:46.829759 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:46.829761 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:46.829772 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:07:46.829798 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:46.829799 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:46.829809 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:46.829875 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:46.830634 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:46.831189 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:46.831768 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:46.831967 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:46.834030 (+ 2063us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:46.834036 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:46.834036 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:46.834043 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:46.835681 (+ 1638us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:46.836059 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:46.836060 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:46.836061 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:46.836089 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:46.836335 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:46.836336 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:46.836336 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:46.836437 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:46.836526 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:46.836655 (+ 129us) server_negotiation.cc:300] Negotiation successful
1026 15:07:46.836765 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
I20251026 15:07:46.838588 16989 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:07:46.838765 16989 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:07:46.840082 17129 tablet_service.cc:1505] Processing CreateTablet for tablet f3a37e067c904acd89a2aa731020cdb9 (DEFAULT_TABLE table=test-table [id=14d5340c7cb448008e567017f315a248]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:46.840085 17272 tablet_service.cc:1505] Processing CreateTablet for tablet f3a37e067c904acd89a2aa731020cdb9 (DEFAULT_TABLE table=test-table [id=14d5340c7cb448008e567017f315a248]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:46.840175 17415 tablet_service.cc:1505] Processing CreateTablet for tablet f3a37e067c904acd89a2aa731020cdb9 (DEFAULT_TABLE table=test-table [id=14d5340c7cb448008e567017f315a248]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:46.840221 17129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a37e067c904acd89a2aa731020cdb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:46.840221 17272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a37e067c904acd89a2aa731020cdb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:46.840286 17415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a37e067c904acd89a2aa731020cdb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:46.841207 17595 tablet_bootstrap.cc:492] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2: Bootstrap starting.
I20251026 15:07:46.841568 17596 tablet_bootstrap.cc:492] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c: Bootstrap starting.
I20251026 15:07:46.841583 17595 tablet_bootstrap.cc:654] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:46.841814 17597 tablet_bootstrap.cc:492] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6: Bootstrap starting.
I20251026 15:07:46.841971 17596 tablet_bootstrap.cc:654] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:46.842234 17597 tablet_bootstrap.cc:654] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:46.842275 17595 tablet_bootstrap.cc:492] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2: No bootstrap required, opened a new log
I20251026 15:07:46.842321 17595 ts_tablet_manager.cc:1403] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:46.842496 17595 raft_consensus.cc:359] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.842573 17595 raft_consensus.cc:385] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:46.842599 17595 raft_consensus.cc:740] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f9f52d8faf498e8ddb1f470bce5bd2, State: Initialized, Role: FOLLOWER
I20251026 15:07:46.842695 17596 tablet_bootstrap.cc:492] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c: No bootstrap required, opened a new log
I20251026 15:07:46.842720 17595 consensus_queue.cc:260] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.842757 17596 ts_tablet_manager.cc:1403] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:46.842902 17595 ts_tablet_manager.cc:1434] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:46.842934 17597 tablet_bootstrap.cc:492] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6: No bootstrap required, opened a new log
I20251026 15:07:46.842906 17596 raft_consensus.cc:359] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.842974 17596 raft_consensus.cc:385] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:46.843019 17596 raft_consensus.cc:740] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf317d62a3f0473fa8e54db7d365221c, State: Initialized, Role: FOLLOWER
I20251026 15:07:46.843060 17597 ts_tablet_manager.cc:1403] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:46.843072 17596 consensus_queue.cc:260] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.843225 17596 ts_tablet_manager.cc:1434] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:46.843207 17597 raft_consensus.cc:359] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.843269 17597 raft_consensus.cc:385] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:46.843317 17597 raft_consensus.cc:740] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e28fc72acb34dedbad811e3b12702b6, State: Initialized, Role: FOLLOWER
I20251026 15:07:46.843393 17597 consensus_queue.cc:260] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.843529 17597 ts_tablet_manager.cc:1434] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:46.879264 17559 raft_consensus.cc:493] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:07:46.879359 17559 raft_consensus.cc:515] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.879534 17559 leader_election.cc:290] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65:38959), 0e28fc72acb34dedbad811e3b12702b6 (127.14.212.67:41061)
I20251026 15:07:46.879745 17435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3a37e067c904acd89a2aa731020cdb9" candidate_uuid: "bf317d62a3f0473fa8e54db7d365221c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e28fc72acb34dedbad811e3b12702b6" is_pre_election: true
I20251026 15:07:46.879741 17145 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3a37e067c904acd89a2aa731020cdb9" candidate_uuid: "bf317d62a3f0473fa8e54db7d365221c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" is_pre_election: true
I20251026 15:07:46.879828 17435 raft_consensus.cc:2468] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf317d62a3f0473fa8e54db7d365221c in term 0.
I20251026 15:07:46.879837 17145 raft_consensus.cc:2468] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf317d62a3f0473fa8e54db7d365221c in term 0.
I20251026 15:07:46.879997 17224 leader_election.cc:304] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e28fc72acb34dedbad811e3b12702b6, bf317d62a3f0473fa8e54db7d365221c; no voters:
I20251026 15:07:46.880131 17559 raft_consensus.cc:2804] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:07:46.880183 17559 raft_consensus.cc:493] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:07:46.880206 17559 raft_consensus.cc:3060] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:46.880927 17559 raft_consensus.cc:515] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.881045 17559 leader_election.cc:290] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [CANDIDATE]: Term 1 election: Requested vote from peers 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65:38959), 0e28fc72acb34dedbad811e3b12702b6 (127.14.212.67:41061)
I20251026 15:07:46.881210 17145 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3a37e067c904acd89a2aa731020cdb9" candidate_uuid: "bf317d62a3f0473fa8e54db7d365221c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2"
I20251026 15:07:46.881258 17435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3a37e067c904acd89a2aa731020cdb9" candidate_uuid: "bf317d62a3f0473fa8e54db7d365221c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e28fc72acb34dedbad811e3b12702b6"
I20251026 15:07:46.881289 17145 raft_consensus.cc:3060] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:46.881319 17435 raft_consensus.cc:3060] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:46.881907 17435 raft_consensus.cc:2468] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf317d62a3f0473fa8e54db7d365221c in term 1.
I20251026 15:07:46.881918 17145 raft_consensus.cc:2468] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf317d62a3f0473fa8e54db7d365221c in term 1.
I20251026 15:07:46.882084 17224 leader_election.cc:304] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e28fc72acb34dedbad811e3b12702b6, bf317d62a3f0473fa8e54db7d365221c; no voters:
I20251026 15:07:46.882180 17559 raft_consensus.cc:2804] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:46.882246 17559 raft_consensus.cc:697] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [term 1 LEADER]: Becoming Leader. State: Replica: bf317d62a3f0473fa8e54db7d365221c, State: Running, Role: LEADER
I20251026 15:07:46.882298 17559 consensus_queue.cc:237] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.882956 16987 catalog_manager.cc:5649] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c reported cstate change: term changed from 0 to 1, leader changed from <none> to bf317d62a3f0473fa8e54db7d365221c (127.14.212.66). New cstate: current_term: 1 leader_uuid: "bf317d62a3f0473fa8e54db7d365221c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } health_report { overall_health: UNKNOWN } } }
I20251026 15:07:46.892158 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:46.890122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.14.212.66:44241)
1026 15:07:46.890260 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:46.890262 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:46.890270 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:07:46.890287 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:46.890288 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:46.890297 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:46.890344 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:46.890644 (+ 300us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:46.891139 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:46.891780 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:46.891901 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:46.891918 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:46.891939 (+ 21us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:46.891987 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:46.892083 (+ 96us) server_negotiation.cc:300] Negotiation successful
1026 15:07:46.892101 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20251026 15:07:46.894272 17435 raft_consensus.cc:1275] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Refusing update from remote peer bf317d62a3f0473fa8e54db7d365221c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:46.894286 17145 raft_consensus.cc:1275] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Refusing update from remote peer bf317d62a3f0473fa8e54db7d365221c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:46.894503 17576 consensus_queue.cc:1048] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [LEADER]: Connected to new peer: Peer: permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:46.894589 17559 consensus_queue.cc:1048] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:46.904793 16989 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35162:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:07:46.904886 16989 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35162:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:07:46.905439 16989 catalog_manager.cc:5936] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606: Sending DeleteTablet for 3 replicas of tablet f3a37e067c904acd89a2aa731020cdb9
I20251026 15:07:46.905652 17129 tablet_service.cc:1552] Processing DeleteTablet for tablet f3a37e067c904acd89a2aa731020cdb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:46 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:47104
I20251026 15:07:46.905668 17415 tablet_service.cc:1552] Processing DeleteTablet for tablet f3a37e067c904acd89a2aa731020cdb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:46 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:43336
I20251026 15:07:46.905676 17272 tablet_service.cc:1552] Processing DeleteTablet for tablet f3a37e067c904acd89a2aa731020cdb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:46 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:39988
I20251026 15:07:46.905752 17584 tablet_replica.cc:333] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2: stopping tablet replica
I20251026 15:07:46.905771 17587 tablet_replica.cc:333] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6: stopping tablet replica
I20251026 15:07:46.905795 17584 raft_consensus.cc:2243] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:07:46.905819 17587 raft_consensus.cc:2243] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:07:46.905829 17584 raft_consensus.cc:2272] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:46.905853 17587 raft_consensus.cc:2272] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:46.905923 17585 tablet_replica.cc:333] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c: stopping tablet replica
I20251026 15:07:46.905969 17585 raft_consensus.cc:2243] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:07:46.905906 16989 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 15:07:46.906021 17585 raft_consensus.cc:2272] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 15:07:46.906047 16989 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:07:46.906175 17584 ts_tablet_manager.cc:1916] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:46.906345 17587 ts_tablet_manager.cc:1916] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:46.906472 17585 ts_tablet_manager.cc:1916] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:46.908041 17585 ts_tablet_manager.cc:1929] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:07:46.908041 17584 ts_tablet_manager.cc:1929] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:07:46.908059 17587 ts_tablet_manager.cc:1929] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:07:46.908102 17587 log.cc:1199] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/f3a37e067c904acd89a2aa731020cdb9
I20251026 15:07:46.908102 17584 log.cc:1199] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-0/wal/wals/f3a37e067c904acd89a2aa731020cdb9
I20251026 15:07:46.908102 17585 log.cc:1199] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/f3a37e067c904acd89a2aa731020cdb9
I20251026 15:07:46.908295 17587 ts_tablet_manager.cc:1950] T f3a37e067c904acd89a2aa731020cdb9 P 0e28fc72acb34dedbad811e3b12702b6: Deleting consensus metadata
I20251026 15:07:46.908330 17585 ts_tablet_manager.cc:1950] T f3a37e067c904acd89a2aa731020cdb9 P bf317d62a3f0473fa8e54db7d365221c: Deleting consensus metadata
I20251026 15:07:46.908330 17584 ts_tablet_manager.cc:1950] T f3a37e067c904acd89a2aa731020cdb9 P 33f9f52d8faf498e8ddb1f470bce5bd2: Deleting consensus metadata
I20251026 15:07:46.908576 16974 catalog_manager.cc:4985] TS 0e28fc72acb34dedbad811e3b12702b6 (127.14.212.67:41061): tablet f3a37e067c904acd89a2aa731020cdb9 (table test-table [id=14d5340c7cb448008e567017f315a248]) successfully deleted
I20251026 15:07:46.908612 16975 catalog_manager.cc:4985] TS 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65:38959): tablet f3a37e067c904acd89a2aa731020cdb9 (table test-table [id=14d5340c7cb448008e567017f315a248]) successfully deleted
I20251026 15:07:46.908793 16975 catalog_manager.cc:4985] TS bf317d62a3f0473fa8e54db7d365221c (127.14.212.66:44241): tablet f3a37e067c904acd89a2aa731020cdb9 (table test-table [id=14d5340c7cb448008e567017f315a248]) successfully deleted
I20251026 15:07:46.909439 17129 tablet_service.cc:1505] Processing CreateTablet for tablet 5c1ee479a9f64c64840dd9f86d519b8f (DEFAULT_TABLE table=test-table [id=ae6e4d457f1e46c5b56fe52a64905ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:46.909482 17415 tablet_service.cc:1505] Processing CreateTablet for tablet 5c1ee479a9f64c64840dd9f86d519b8f (DEFAULT_TABLE table=test-table [id=ae6e4d457f1e46c5b56fe52a64905ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:46.909490 17272 tablet_service.cc:1505] Processing CreateTablet for tablet 5c1ee479a9f64c64840dd9f86d519b8f (DEFAULT_TABLE table=test-table [id=ae6e4d457f1e46c5b56fe52a64905ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:07:46.909555 17129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c1ee479a9f64c64840dd9f86d519b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:46.909560 17415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c1ee479a9f64c64840dd9f86d519b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:46.909601 17272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c1ee479a9f64c64840dd9f86d519b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:46.910441 17596 tablet_bootstrap.cc:492] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c: Bootstrap starting.
I20251026 15:07:46.910766 17595 tablet_bootstrap.cc:492] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2: Bootstrap starting.
I20251026 15:07:46.910769 17597 tablet_bootstrap.cc:492] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6: Bootstrap starting.
I20251026 15:07:46.910794 17596 tablet_bootstrap.cc:654] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:46.911039 17597 tablet_bootstrap.cc:654] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:46.911155 17595 tablet_bootstrap.cc:654] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:46.911393 17596 tablet_bootstrap.cc:492] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c: No bootstrap required, opened a new log
I20251026 15:07:46.911444 17596 ts_tablet_manager.cc:1403] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 15:07:46.911579 17596 raft_consensus.cc:359] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.911643 17596 raft_consensus.cc:385] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:46.911677 17596 raft_consensus.cc:740] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf317d62a3f0473fa8e54db7d365221c, State: Initialized, Role: FOLLOWER
I20251026 15:07:46.911728 17596 consensus_queue.cc:260] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.911773 17595 tablet_bootstrap.cc:492] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2: No bootstrap required, opened a new log
I20251026 15:07:46.911810 17595 ts_tablet_manager.cc:1403] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 15:07:46.911826 17596 ts_tablet_manager.cc:1434] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:46.911962 17595 raft_consensus.cc:359] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.912024 17595 raft_consensus.cc:385] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:46.912045 17595 raft_consensus.cc:740] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f9f52d8faf498e8ddb1f470bce5bd2, State: Initialized, Role: FOLLOWER
I20251026 15:07:46.912096 17595 consensus_queue.cc:260] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.912195 17595 ts_tablet_manager.cc:1434] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:46.912447 17597 tablet_bootstrap.cc:492] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6: No bootstrap required, opened a new log
I20251026 15:07:46.912528 17597 ts_tablet_manager.cc:1403] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20251026 15:07:46.912657 17597 raft_consensus.cc:359] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.912732 17597 raft_consensus.cc:385] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:46.912753 17597 raft_consensus.cc:740] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e28fc72acb34dedbad811e3b12702b6, State: Initialized, Role: FOLLOWER
I20251026 15:07:46.912796 17597 consensus_queue.cc:260] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:46.912885 17597 ts_tablet_manager.cc:1434] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:47.153951 17601 raft_consensus.cc:493] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:07:47.154059 17601 raft_consensus.cc:515] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:47.154326 17601 leader_election.cc:290] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf317d62a3f0473fa8e54db7d365221c (127.14.212.66:44241), 0e28fc72acb34dedbad811e3b12702b6 (127.14.212.67:41061)
I20251026 15:07:47.157490 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:47.154533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:44241 (local address 127.14.212.65:54367)
1026 15:07:47.154665 (+ 132us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:47.154679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:47.154806 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:47.154969 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:47.154970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:47.154974 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:47.155289 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:47.155291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.156234 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:47.156235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:47.157207 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:47.157208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.157309 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:47.157312 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:07:47.157348 (+ 36us) client_negotiation.cc:240] Negotiation successful
1026 15:07:47.157377 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20251026 15:07:47.157490 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:47.154557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:41061 (local address 127.14.212.65:34443)
1026 15:07:47.154740 (+ 183us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:47.154748 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:47.154806 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:47.154980 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:47.154981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:47.154982 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:07:47.155289 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:47.155291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.156234 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:47.156235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:47.157196 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:47.157199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.157292 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:47.157297 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:07:47.157338 (+ 41us) client_negotiation.cc:240] Negotiation successful
1026 15:07:47.157377 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":107,"threads_started":1}
I20251026 15:07:47.157908 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:47.154628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:34443 (local address 127.14.212.67:41061)
1026 15:07:47.154788 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:47.154791 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:47.154816 (+ 25us) server_negotiation.cc:408] Connection header received
1026 15:07:47.154872 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:47.154873 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:47.154886 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:47.154955 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:47.155400 (+ 445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.156148 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:47.157307 (+ 1159us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.157784 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:47.157790 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:47.157814 (+ 24us) server_negotiation.cc:300] Negotiation successful
1026 15:07:47.157843 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20251026 15:07:47.157907 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:47.154679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:54367 (local address 127.14.212.66:44241)
1026 15:07:47.154727 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:47.154729 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:47.154815 (+ 86us) server_negotiation.cc:408] Connection header received
1026 15:07:47.154872 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:47.154873 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:47.154885 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:47.154955 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:07:47.155359 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.156131 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:47.157304 (+ 1173us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.157782 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:47.157790 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:47.157813 (+ 23us) server_negotiation.cc:300] Negotiation successful
1026 15:07:47.157843 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251026 15:07:47.158097 17435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c1ee479a9f64c64840dd9f86d519b8f" candidate_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e28fc72acb34dedbad811e3b12702b6" is_pre_election: true
I20251026 15:07:47.158097 17292 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c1ee479a9f64c64840dd9f86d519b8f" candidate_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf317d62a3f0473fa8e54db7d365221c" is_pre_election: true
I20251026 15:07:47.158182 17435 raft_consensus.cc:2468] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33f9f52d8faf498e8ddb1f470bce5bd2 in term 0.
I20251026 15:07:47.158182 17292 raft_consensus.cc:2468] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33f9f52d8faf498e8ddb1f470bce5bd2 in term 0.
I20251026 15:07:47.158416 17081 leader_election.cc:304] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e28fc72acb34dedbad811e3b12702b6, 33f9f52d8faf498e8ddb1f470bce5bd2; no voters:
I20251026 15:07:47.158545 17601 raft_consensus.cc:2804] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:07:47.158586 17601 raft_consensus.cc:493] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:07:47.158613 17601 raft_consensus.cc:3060] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:47.159221 17601 raft_consensus.cc:515] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:47.159341 17601 leader_election.cc:290] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [CANDIDATE]: Term 1 election: Requested vote from peers bf317d62a3f0473fa8e54db7d365221c (127.14.212.66:44241), 0e28fc72acb34dedbad811e3b12702b6 (127.14.212.67:41061)
I20251026 15:07:47.159627 17292 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c1ee479a9f64c64840dd9f86d519b8f" candidate_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf317d62a3f0473fa8e54db7d365221c"
I20251026 15:07:47.159618 17435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c1ee479a9f64c64840dd9f86d519b8f" candidate_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e28fc72acb34dedbad811e3b12702b6"
I20251026 15:07:47.159704 17435 raft_consensus.cc:3060] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:47.159704 17292 raft_consensus.cc:3060] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:47.160449 17292 raft_consensus.cc:2468] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33f9f52d8faf498e8ddb1f470bce5bd2 in term 1.
I20251026 15:07:47.160449 17435 raft_consensus.cc:2468] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33f9f52d8faf498e8ddb1f470bce5bd2 in term 1.
I20251026 15:07:47.160666 17081 leader_election.cc:304] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e28fc72acb34dedbad811e3b12702b6, 33f9f52d8faf498e8ddb1f470bce5bd2; no voters:
I20251026 15:07:47.160799 17601 raft_consensus.cc:2804] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:47.160945 17601 raft_consensus.cc:697] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [term 1 LEADER]: Becoming Leader. State: Replica: 33f9f52d8faf498e8ddb1f470bce5bd2, State: Running, Role: LEADER
I20251026 15:07:47.161015 17601 consensus_queue.cc:237] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } }
I20251026 15:07:47.161731 16989 catalog_manager.cc:5649] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65). New cstate: current_term: 1 leader_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f9f52d8faf498e8ddb1f470bce5bd2" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 } health_report { overall_health: UNKNOWN } } }
I20251026 15:07:47.199890 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:47.197397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.14.212.65:38959)
1026 15:07:47.197580 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:47.197582 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:47.197592 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:07:47.197630 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:47.197631 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:47.197649 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:47.197713 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:47.197991 (+ 278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.198575 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:47.199265 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.199451 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:47.199487 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:47.199587 (+ 100us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:47.199662 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:47.199753 (+ 91us) server_negotiation.cc:300] Negotiation successful
1026 15:07:47.199780 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
I20251026 15:07:47.204082 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:47.202308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:43233 (local address 127.14.212.67:41061)
1026 15:07:47.202381 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:47.202383 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:47.202411 (+ 28us) server_negotiation.cc:408] Connection header received
1026 15:07:47.202440 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:47.202441 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:47.202450 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:47.202503 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:47.202739 (+ 236us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.203216 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:47.203756 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.203869 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:47.203885 (+ 16us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:47.203905 (+ 20us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:47.203973 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:47.204003 (+ 30us) server_negotiation.cc:300] Negotiation successful
1026 15:07:47.204025 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251026 15:07:47.204098 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:47.202127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:41061 (local address 127.14.212.65:43233)
1026 15:07:47.202294 (+ 167us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:47.202303 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:47.202347 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:47.202510 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:47.202511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:47.202512 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 15:07:47.202679 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:47.202680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.203283 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:47.203284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:47.203693 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:47.203694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.203765 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:47.203779 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 15:07:47.203965 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 15:07:47.203966 (+ 1us) client_negotiation.cc:769] Sending connection context
1026 15:07:47.204000 (+ 34us) client_negotiation.cc:240] Negotiation successful
1026 15:07:47.204036 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20251026 15:07:47.206102 17435 raft_consensus.cc:1275] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Refusing update from remote peer 33f9f52d8faf498e8ddb1f470bce5bd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:47.206092 17292 raft_consensus.cc:1275] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Refusing update from remote peer 33f9f52d8faf498e8ddb1f470bce5bd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:07:47.206336 17601 consensus_queue.cc:1048] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf317d62a3f0473fa8e54db7d365221c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:47.206491 17615 consensus_queue.cc:1048] T 5c1ee479a9f64c64840dd9f86d519b8f P 33f9f52d8faf498e8ddb1f470bce5bd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e28fc72acb34dedbad811e3b12702b6" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 41061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:07:47.213527 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:47.211462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:38959 (local address 127.14.212.67:36369)
1026 15:07:47.211604 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:47.211615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:47.211687 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:47.211832 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:47.211833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:47.211834 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 15:07:47.212006 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:47.212008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.212571 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:47.212572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:47.213045 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:47.213048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.213112 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:47.213118 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 15:07:47.213365 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 15:07:47.213367 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:07:47.213412 (+ 45us) client_negotiation.cc:240] Negotiation successful
1026 15:07:47.213462 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":97,"threads_started":1}
I20251026 15:07:47.213704 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:47.211499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:36369 (local address 127.14.212.65:38959)
1026 15:07:47.211544 (+ 45us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:47.211547 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:47.211748 (+ 201us) server_negotiation.cc:408] Connection header received
1026 15:07:47.211769 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:47.211769 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:47.211781 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:47.211828 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:07:47.212060 (+ 232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.212514 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:47.213108 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:47.213266 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:47.213283 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:07:47.213305 (+ 22us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:07:47.213354 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:47.213616 (+ 262us) server_negotiation.cc:300] Negotiation successful
1026 15:07:47.213644 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20251026 15:07:47.214015 17567 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251026 15:07:48.738713 16988 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35162:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:07:48.738814 16988 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35162:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:07:48.739840 16988 catalog_manager.cc:5936] T 00000000000000000000000000000000 P be14f780bfe74bdc98766fe9ab0ab606: Sending DeleteTablet for 3 replicas of tablet 5c1ee479a9f64c64840dd9f86d519b8f
I20251026 15:07:48.740106 17129 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c1ee479a9f64c64840dd9f86d519b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:48 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:47104
I20251026 15:07:48.740180 17415 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c1ee479a9f64c64840dd9f86d519b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:48 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:43336
I20251026 15:07:48.740191 17272 tablet_service.cc:1552] Processing DeleteTablet for tablet 5c1ee479a9f64c64840dd9f86d519b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:07:48 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:39988
I20251026 15:07:48.740278 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 17064
I20251026 15:07:48.741333 17649 tablet_replica.cc:333] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c: stopping tablet replica
I20251026 15:07:48.741415 17649 raft_consensus.cc:2243] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:07:48.741468 17649 raft_consensus.cc:2272] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:48.741601 17650 tablet_replica.cc:333] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6: stopping tablet replica
I20251026 15:07:48.741696 17650 raft_consensus.cc:2243] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:07:48.741747 17650 raft_consensus.cc:2272] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:07:48.741745 17649 ts_tablet_manager.cc:1916] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:48.742010 17650 ts_tablet_manager.cc:1916] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:07:48.743496 17649 ts_tablet_manager.cc:1929] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251026 15:07:48.743568 17649 log.cc:1199] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/5c1ee479a9f64c64840dd9f86d519b8f
I20251026 15:07:48.743686 17650 ts_tablet_manager.cc:1929] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251026 15:07:48.743743 17650 log.cc:1199] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/5c1ee479a9f64c64840dd9f86d519b8f
I20251026 15:07:48.743850 17649 ts_tablet_manager.cc:1950] T 5c1ee479a9f64c64840dd9f86d519b8f P bf317d62a3f0473fa8e54db7d365221c: Deleting consensus metadata
I20251026 15:07:48.744065 17650 ts_tablet_manager.cc:1950] T 5c1ee479a9f64c64840dd9f86d519b8f P 0e28fc72acb34dedbad811e3b12702b6: Deleting consensus metadata
I20251026 15:07:48.744205 16975 catalog_manager.cc:4985] TS bf317d62a3f0473fa8e54db7d365221c (127.14.212.66:44241): tablet 5c1ee479a9f64c64840dd9f86d519b8f (table test-table [id=ae6e4d457f1e46c5b56fe52a64905ef7]) successfully deleted
I20251026 15:07:48.744638 16974 catalog_manager.cc:4985] TS 0e28fc72acb34dedbad811e3b12702b6 (127.14.212.67:41061): tablet 5c1ee479a9f64c64840dd9f86d519b8f (table test-table [id=ae6e4d457f1e46c5b56fe52a64905ef7]) successfully deleted
W20251026 15:07:48.747139 16975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:38959 (error 108)
I20251026 15:07:48.747385 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 17207
I20251026 15:07:48.748132 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:48.747516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:38959 (local address 127.0.0.1:47128)
1026 15:07:48.748037 (+ 521us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:48.748081 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:38959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":58,"threads_started":1}
W20251026 15:07:48.749094 16975 catalog_manager.cc:4712] TS 33f9f52d8faf498e8ddb1f470bce5bd2 (127.14.212.65:38959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c1ee479a9f64c64840dd9f86d519b8f: Network error: Client connection negotiation failed: client connection to 127.14.212.65:38959: connect: Connection refused (error 111)
I20251026 15:07:48.753314 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 17350
I20251026 15:07:48.759011 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 16957
2025-10-26T15:07:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6843 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:07:48 dist-test-slave-8pgz krb5kdc[17655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:07:48 dist-test-slave-8pgz krb5kdc[17655](info): set up 2 sockets
Oct 26 15:07:48 dist-test-slave-8pgz krb5kdc[17655](info): commencing operation
krb5kdc: starting...
W20251026 15:07:50.775797 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.995s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:07:50 dist-test-slave-8pgz krb5kdc[17655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:07:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:07:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:50.915349 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:41969
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:41969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:50.996250 17672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:50.996588 17672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:50.996649 17672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:50.998538 17672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:07:50.998589 17672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:50.998605 17672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:50.998616 17672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:07:50.998631 17672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:07:51.000322 17672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:41969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:41969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:51.000536 17672 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:51.000733 17672 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:51.003489 17680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:51.003491 17677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:51.003505 17672 server_base.cc:1047] running on GCE node
W20251026 15:07:51.003501 17678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:51.004122 17672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:51.004349 17672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:51.005506 17672 hybrid_clock.cc:648] HybridClock initialized: now 1761491271005478 us; error 45 us; skew 500 ppm
Oct 26 15:07:51 dist-test-slave-8pgz krb5kdc[17655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491271, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:51.007964 17672 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:07:51.008498 17672 webserver.cc:492] Webserver started at http://127.14.212.126:38889/ using document root <none> and password file <none>
I20251026 15:07:51.008670 17672 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:51.008702 17672 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:51.008791 17672 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:51.009704 17672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "2d25bdbf290347089b58b04b82ddb5ee"
format_stamp: "Formatted at 2025-10-26 15:07:51 on dist-test-slave-8pgz"
server_key: "a3c66117a825574cfe8b704ae8e3c153"
server_key_iv: "b27264d2a898b5d7ef3409a885a49a4a"
server_key_version: "encryptionkey@0"
I20251026 15:07:51.009968 17672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "2d25bdbf290347089b58b04b82ddb5ee"
format_stamp: "Formatted at 2025-10-26 15:07:51 on dist-test-slave-8pgz"
server_key: "a3c66117a825574cfe8b704ae8e3c153"
server_key_iv: "b27264d2a898b5d7ef3409a885a49a4a"
server_key_version: "encryptionkey@0"
I20251026 15:07:51.011220 17672 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:07:51.011970 17687 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:51.012203 17672 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:51.012286 17672 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "2d25bdbf290347089b58b04b82ddb5ee"
format_stamp: "Formatted at 2025-10-26 15:07:51 on dist-test-slave-8pgz"
server_key: "a3c66117a825574cfe8b704ae8e3c153"
server_key_iv: "b27264d2a898b5d7ef3409a885a49a4a"
server_key_version: "encryptionkey@0"
I20251026 15:07:51.012354 17672 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:51.041318 17672 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:51.043401 17672 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:51.043526 17672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:51.047428 17672 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:41969
I20251026 15:07:51.047461 17739 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:41969 every 8 connection(s)
I20251026 15:07:51.047765 17672 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:07:51.048341 17740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:51.050065 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 17672
I20251026 15:07:51.050163 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:07:51.050326 15185 external_mini_cluster.cc:1468] Setting key 89ec4b3d820f7d66d4a15a60c2c9eb79
I20251026 15:07:51.051124 17740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee: Bootstrap starting.
I20251026 15:07:51.051764 17740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:51.052053 17740 log.cc:826] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:51.052712 17740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee: No bootstrap required, opened a new log
I20251026 15:07:51.053959 17740 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25bdbf290347089b58b04b82ddb5ee" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41969 } }
I20251026 15:07:51.054128 17740 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:51.054170 17740 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d25bdbf290347089b58b04b82ddb5ee, State: Initialized, Role: FOLLOWER
I20251026 15:07:51.054314 17740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25bdbf290347089b58b04b82ddb5ee" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41969 } }
I20251026 15:07:51.054414 17740 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:07:51.054461 17740 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:07:51.054507 17740 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:51.055308 17740 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25bdbf290347089b58b04b82ddb5ee" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41969 } }
I20251026 15:07:51.055425 17740 leader_election.cc:304] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d25bdbf290347089b58b04b82ddb5ee; no voters:
I20251026 15:07:51.055572 17740 leader_election.cc:290] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:07:51.055627 17745 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:51.055800 17745 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [term 1 LEADER]: Becoming Leader. State: Replica: 2d25bdbf290347089b58b04b82ddb5ee, State: Running, Role: LEADER
I20251026 15:07:51.055809 17740 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [sys.catalog]: configured and running, proceeding with master startup.
Oct 26 15:07:51 dist-test-slave-8pgz krb5kdc[17655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:51.056582 17745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25bdbf290347089b58b04b82ddb5ee" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41969 } }
I20251026 15:07:51.057332 17747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d25bdbf290347089b58b04b82ddb5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25bdbf290347089b58b04b82ddb5ee" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41969 } } }
I20251026 15:07:51.057361 17746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d25bdbf290347089b58b04b82ddb5ee. Latest consensus state: current_term: 1 leader_uuid: "2d25bdbf290347089b58b04b82ddb5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25bdbf290347089b58b04b82ddb5ee" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41969 } } }
I20251026 15:07:51.057637 17747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [sys.catalog]: This master's current role is: LEADER
I20251026 15:07:51.057648 17746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee [sys.catalog]: This master's current role is: LEADER
W20251026 15:07:51.058369 17759 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:07:51.058439 17759 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:07:51.058503 17760 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:07:51.058696 17760 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:07:51.060214 17760 catalog_manager.cc:1357] Generated new cluster ID: 57c8b0585b754a43aef01e52232e20c3
I20251026 15:07:51.060302 17760 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:07:51.060501 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:51.050870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.14.212.126:41969)
1026 15:07:51.051021 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:51.051031 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:51.051049 (+ 18us) server_negotiation.cc:408] Connection header received
1026 15:07:51.051362 (+ 313us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:51.051368 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:51.051411 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:51.051516 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:51.052483 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.053232 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:51.054000 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.054221 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:51.056767 (+ 2546us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:51.056774 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:51.056775 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:51.056815 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:51.059000 (+ 2185us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:51.059363 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:51.059366 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:51.059366 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:51.059426 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:51.059608 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:51.059609 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:51.059609 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:51.059901 (+ 292us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:51.060046 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:51.060196 (+ 150us) server_negotiation.cc:300] Negotiation successful
1026 15:07:51.060318 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20251026 15:07:51.076220 17760 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:07:51.076814 17760 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:07:51.079964 17760 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 2d25bdbf290347089b58b04b82ddb5ee: Generated new TSK 0
I20251026 15:07:51.080142 17760 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:51.135138 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41969
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:51.215204 17768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:51.215360 17768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:51.215384 17768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:51.216825 17768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:51.216871 17768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:51.216897 17768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:07:51.218466 17768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:51.218665 17768 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:51.218852 17768 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:07:51.221371 17768 server_base.cc:1047] running on GCE node
W20251026 15:07:51.221333 17774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:51.221339 17773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:51.221408 17776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:51.221738 17768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:51.221920 17768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:51.223053 17768 hybrid_clock.cc:648] HybridClock initialized: now 1761491271223047 us; error 27 us; skew 500 ppm
Oct 26 15:07:51 dist-test-slave-8pgz krb5kdc[17655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491271, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:51.225226 17768 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:07:51.225680 17768 webserver.cc:492] Webserver started at http://127.14.212.65:33497/ using document root <none> and password file <none>
I20251026 15:07:51.225836 17768 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:51.225867 17768 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:51.225952 17768 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:51.226897 17768 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "abdb5333383540d5b5bdb3b67b37c107"
format_stamp: "Formatted at 2025-10-26 15:07:51 on dist-test-slave-8pgz"
server_key: "7fd0df814ac49cd80076012bb5897e8a"
server_key_iv: "24e6e04e2446c072576562633a33b142"
server_key_version: "encryptionkey@0"
I20251026 15:07:51.227165 17768 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "abdb5333383540d5b5bdb3b67b37c107"
format_stamp: "Formatted at 2025-10-26 15:07:51 on dist-test-slave-8pgz"
server_key: "7fd0df814ac49cd80076012bb5897e8a"
server_key_iv: "24e6e04e2446c072576562633a33b142"
server_key_version: "encryptionkey@0"
I20251026 15:07:51.228282 17768 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:07:51.228909 17783 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:51.229151 17768 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:51.229226 17768 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "abdb5333383540d5b5bdb3b67b37c107"
format_stamp: "Formatted at 2025-10-26 15:07:51 on dist-test-slave-8pgz"
server_key: "7fd0df814ac49cd80076012bb5897e8a"
server_key_iv: "24e6e04e2446c072576562633a33b142"
server_key_version: "encryptionkey@0"
I20251026 15:07:51.229281 17768 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:51.239342 17768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:51.241053 17768 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:51.241154 17768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:51.241317 17768 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:07:51.241611 17768 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:51.241642 17768 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:51.241664 17768 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:51.241690 17768 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:51.247146 17768 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:43951
I20251026 15:07:51.247224 17897 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:43951 every 8 connection(s)
I20251026 15:07:51.247469 17768 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:07:51.249511 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 17768
I20251026 15:07:51.249610 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:07:51.249791 15185 external_mini_cluster.cc:1468] Setting key 55faf5ab60eeb6f22a5c2b019fa354a0
Oct 26 15:07:51 dist-test-slave-8pgz krb5kdc[17655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491271, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:51.256569 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:51.248084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:32929 (local address 127.14.212.126:41969)
1026 15:07:51.248165 (+ 81us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:51.248167 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:51.248267 (+ 100us) server_negotiation.cc:408] Connection header received
1026 15:07:51.248711 (+ 444us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:51.248712 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:51.248722 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:51.248769 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:51.249564 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.250183 (+ 619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:51.251566 (+ 1383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.251696 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:51.253796 (+ 2100us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:51.253800 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:51.253800 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:51.253807 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:51.255390 (+ 1583us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:51.255748 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:51.255749 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:51.255750 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:51.255774 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:51.255972 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:51.255973 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:51.255973 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:51.256067 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:51.256128 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:51.256373 (+ 245us) server_negotiation.cc:300] Negotiation successful
1026 15:07:51.256476 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251026 15:07:51.256667 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:51.248028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:41969 (local address 127.14.212.65:32929)
1026 15:07:51.248153 (+ 125us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:51.248177 (+ 24us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:51.248633 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:51.248825 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:51.248828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:51.249140 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:51.249463 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:51.249468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.250256 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:51.250257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:51.251480 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:51.251483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.251626 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:51.252284 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:51.252302 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:51.253698 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:51.255472 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:51.255478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:51.255481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:51.255680 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:51.255852 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:51.255853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:51.255854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:51.255915 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:51.256120 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:51.256125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:51.256251 (+ 126us) client_negotiation.cc:769] Sending connection context
1026 15:07:51.256349 (+ 98us) client_negotiation.cc:240] Negotiation successful
1026 15:07:51.256482 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20251026 15:07:51.257126 17898 heartbeater.cc:344] Connected to a master server at 127.14.212.126:41969
I20251026 15:07:51.257221 17898 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
I20251026 15:07:51.257449 17898 heartbeater.cc:507] Master 127.14.212.126:41969 requested a full tablet report, sending...
I20251026 15:07:51.257920 17704 ts_manager.cc:194] Registered new tserver with Master: abdb5333383540d5b5bdb3b67b37c107 (127.14.212.65:43951)
I20251026 15:07:51.258649 17704 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:32929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:51.297914 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41969
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:51.376034 17905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:51.376194 17905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:51.376215 17905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:51.377861 17905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:51.377904 17905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:51.377929 17905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:07:51.379386 17905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:51.379568 17905 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:51.379714 17905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:51.382048 17910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:51.382302 17911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:51.382381 17913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:51.382447 17905 server_base.cc:1047] running on GCE node
I20251026 15:07:51.382608 17905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:51.382776 17905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:51.383903 17905 hybrid_clock.cc:648] HybridClock initialized: now 1761491271383896 us; error 25 us; skew 500 ppm
Oct 26 15:07:51 dist-test-slave-8pgz krb5kdc[17655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491271, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:51.386127 17905 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:07:51.386682 17905 webserver.cc:492] Webserver started at http://127.14.212.66:38881/ using document root <none> and password file <none>
I20251026 15:07:51.386847 17905 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:51.386878 17905 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:51.386960 17905 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:51.387874 17905 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "85dd9160e36a448fa349647804020a5a"
format_stamp: "Formatted at 2025-10-26 15:07:51 on dist-test-slave-8pgz"
server_key: "f29f67d106f6490a2f67289cfe01bd56"
server_key_iv: "9d70a3d9298cff804715a8bb61d80b29"
server_key_version: "encryptionkey@0"
I20251026 15:07:51.388144 17905 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "85dd9160e36a448fa349647804020a5a"
format_stamp: "Formatted at 2025-10-26 15:07:51 on dist-test-slave-8pgz"
server_key: "f29f67d106f6490a2f67289cfe01bd56"
server_key_iv: "9d70a3d9298cff804715a8bb61d80b29"
server_key_version: "encryptionkey@0"
I20251026 15:07:51.389282 17905 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:07:51.389997 17920 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:51.390259 17905 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:51.390331 17905 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "85dd9160e36a448fa349647804020a5a"
format_stamp: "Formatted at 2025-10-26 15:07:51 on dist-test-slave-8pgz"
server_key: "f29f67d106f6490a2f67289cfe01bd56"
server_key_iv: "9d70a3d9298cff804715a8bb61d80b29"
server_key_version: "encryptionkey@0"
I20251026 15:07:51.390383 17905 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:51.404986 17905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:51.406828 17905 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:51.406929 17905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:51.407099 17905 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:07:51.407409 17905 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:51.407442 17905 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:51.407475 17905 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:51.407496 17905 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:51.413127 17905 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:40565
I20251026 15:07:51.413167 18034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:40565 every 8 connection(s)
I20251026 15:07:51.413447 17905 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
Oct 26 15:07:51 dist-test-slave-8pgz krb5kdc[17655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491271, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:51.421442 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:51.414132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:49463 (local address 127.14.212.126:41969)
1026 15:07:51.414201 (+ 69us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:51.414203 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:51.414316 (+ 113us) server_negotiation.cc:408] Connection header received
1026 15:07:51.414865 (+ 549us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:51.414867 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:51.414876 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:51.414930 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:51.415679 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.416168 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:51.417101 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.417215 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:51.419120 (+ 1905us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:51.419125 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:51.419125 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:51.419131 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:51.420443 (+ 1312us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:51.420773 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:51.420773 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:51.420774 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:51.420794 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:51.420959 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:51.420960 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:51.420960 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:51.421056 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:51.421103 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:51.421308 (+ 205us) server_negotiation.cc:300] Negotiation successful
1026 15:07:51.421376 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251026 15:07:51.421545 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:51.414097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:41969 (local address 127.14.212.66:49463)
1026 15:07:51.414229 (+ 132us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:51.414254 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:51.414740 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:51.414991 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:51.414994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:51.415313 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:51.415580 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:51.415583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.416244 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:51.416245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:51.417040 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:51.417041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.417146 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:51.417628 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:51.417641 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:51.419029 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:51.420515 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:51.420520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:51.420522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:51.420714 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:51.420860 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:51.420860 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:51.420861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:51.420916 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:51.421100 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:51.421102 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:51.421211 (+ 109us) client_negotiation.cc:769] Sending connection context
1026 15:07:51.421294 (+ 83us) client_negotiation.cc:240] Negotiation successful
1026 15:07:51.421396 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20251026 15:07:51.421996 18035 heartbeater.cc:344] Connected to a master server at 127.14.212.126:41969
I20251026 15:07:51.422079 18035 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:51.422261 18035 heartbeater.cc:507] Master 127.14.212.126:41969 requested a full tablet report, sending...
I20251026 15:07:51.422515 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 17905
I20251026 15:07:51.422613 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:07:51.422641 17704 ts_manager.cc:194] Registered new tserver with Master: 85dd9160e36a448fa349647804020a5a (127.14.212.66:40565)
I20251026 15:07:51.422781 15185 external_mini_cluster.cc:1468] Setting key d8b54dfb2cdc6320054d02b6d42b977c
I20251026 15:07:51.423143 17704 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:49463
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:51.467412 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41969
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:51.547822 18042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:51.547997 18042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:51.548022 18042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:51.549463 18042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:51.549511 18042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:51.549538 18042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:07:51.551164 18042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:51.551398 18042 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:51.551600 18042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:51.554175 18047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:51.554265 18042 server_base.cc:1047] running on GCE node
W20251026 15:07:51.554184 18048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:51.554186 18050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:51.554559 18042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:51.554762 18042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:51.555949 18042 hybrid_clock.cc:648] HybridClock initialized: now 1761491271555936 us; error 31 us; skew 500 ppm
Oct 26 15:07:51 dist-test-slave-8pgz krb5kdc[17655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491271, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:51.558346 18042 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:07:51.558916 18042 webserver.cc:492] Webserver started at http://127.14.212.67:35613/ using document root <none> and password file <none>
I20251026 15:07:51.559103 18042 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:51.559145 18042 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:51.559260 18042 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:51.560238 18042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "15703add258a4f4a8e0773b3da2da151"
format_stamp: "Formatted at 2025-10-26 15:07:51 on dist-test-slave-8pgz"
server_key: "75b4a4c7588a5a78a1886d7c5b4ace22"
server_key_iv: "d29aea7f9cad5381c54c48622b2333de"
server_key_version: "encryptionkey@0"
I20251026 15:07:51.560554 18042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "15703add258a4f4a8e0773b3da2da151"
format_stamp: "Formatted at 2025-10-26 15:07:51 on dist-test-slave-8pgz"
server_key: "75b4a4c7588a5a78a1886d7c5b4ace22"
server_key_iv: "d29aea7f9cad5381c54c48622b2333de"
server_key_version: "encryptionkey@0"
I20251026 15:07:51.561743 18042 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:07:51.562373 18057 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:51.562557 18042 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:51.562631 18042 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "15703add258a4f4a8e0773b3da2da151"
format_stamp: "Formatted at 2025-10-26 15:07:51 on dist-test-slave-8pgz"
server_key: "75b4a4c7588a5a78a1886d7c5b4ace22"
server_key_iv: "d29aea7f9cad5381c54c48622b2333de"
server_key_version: "encryptionkey@0"
I20251026 15:07:51.562696 18042 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:51.569187 18042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:51.571255 18042 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:51.571399 18042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:51.571617 18042 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:07:51.571933 18042 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:51.571965 18042 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:51.572001 18042 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:51.572021 18042 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:51.577698 18042 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:39189
I20251026 15:07:51.577732 18172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:39189 every 8 connection(s)
I20251026 15:07:51.578027 18042 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:07:51.581665 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 18042
I20251026 15:07:51.581761 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:07:51.581939 15185 external_mini_cluster.cc:1468] Setting key 5f9e8eed72a070528ba247567160e408
Oct 26 15:07:51 dist-test-slave-8pgz krb5kdc[17655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491271, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:51.586714 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:51.578688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:56109 (local address 127.14.212.126:41969)
1026 15:07:51.578802 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:51.578804 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:51.578983 (+ 179us) server_negotiation.cc:408] Connection header received
1026 15:07:51.579468 (+ 485us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:51.579469 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:51.579479 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:51.579531 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:51.580383 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.580967 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:51.582183 (+ 1216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.582350 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:51.584337 (+ 1987us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:51.584340 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:51.584340 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:51.584347 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:51.585611 (+ 1264us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:51.585993 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:51.585993 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:51.585993 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:51.586012 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:51.586152 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:51.586152 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:51.586153 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:51.586232 (+ 79us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:51.586278 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:51.586575 (+ 297us) server_negotiation.cc:300] Negotiation successful
1026 15:07:51.586648 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20251026 15:07:51.586930 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:51.578631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:41969 (local address 127.14.212.67:56109)
1026 15:07:51.578872 (+ 241us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:51.578898 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:51.579379 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:51.579579 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:51.579582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:51.579975 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:51.580298 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:51.580302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.581069 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:51.581070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:51.582107 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:51.582110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:51.582246 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:51.582935 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:51.582957 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:51.584263 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:51.585690 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:51.585695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:51.585697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:51.585933 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:51.586061 (+ 128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:51.586062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:51.586062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:51.586112 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:51.586283 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:51.586287 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:51.586464 (+ 177us) client_negotiation.cc:769] Sending connection context
1026 15:07:51.586558 (+ 94us) client_negotiation.cc:240] Negotiation successful
1026 15:07:51.586682 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":75,"threads_started":1}
I20251026 15:07:51.587448 18173 heartbeater.cc:344] Connected to a master server at 127.14.212.126:41969
I20251026 15:07:51.587549 18173 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:51.587755 18173 heartbeater.cc:507] Master 127.14.212.126:41969 requested a full tablet report, sending...
I20251026 15:07:51.588111 17703 ts_manager.cc:194] Registered new tserver with Master: 15703add258a4f4a8e0773b3da2da151 (127.14.212.67:39189)
I20251026 15:07:51.588500 17703 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:56109
I20251026 15:07:51.593663 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251026 15:07:51.604131 17743 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:07:51.603202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.14.212.126:41969)
1026 15:07:51.603279 (+ 77us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:51.603282 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:51.603292 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:07:51.603326 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:51.603327 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:51.603340 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:51.603404 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:51.604070 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50716: BlockingRecv error: recv got EOF from 127.0.0.1:50716 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20251026 15:07:51.605322 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 17768
I20251026 15:07:51.610162 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 17905
I20251026 15:07:51.614756 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 18042
I20251026 15:07:51.619454 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 17672
2025-10-26T15:07:51Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2859 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:07:51 dist-test-slave-8pgz krb5kdc[18185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:07:51 dist-test-slave-8pgz krb5kdc[18185](info): set up 2 sockets
Oct 26 15:07:51 dist-test-slave-8pgz krb5kdc[18185](info): commencing operation
krb5kdc: starting...
W20251026 15:07:53.623880 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.984s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:07:53 dist-test-slave-8pgz krb5kdc[18185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:07:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:07:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:53.762218 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:41479
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:41479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:53.843875 18201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:53.844061 18201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:53.844084 18201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:53.845487 18201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:07:53.845533 18201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:53.845547 18201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:53.845561 18201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:07:53.845572 18201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:07:53.847216 18201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:41479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:41479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:53.847437 18201 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:53.847637 18201 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:53.850077 18207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:53.850066 18209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:53.850066 18206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:53.850406 18201 server_base.cc:1047] running on GCE node
I20251026 15:07:53.850596 18201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:53.850793 18201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:53.851934 18201 hybrid_clock.cc:648] HybridClock initialized: now 1761491273851924 us; error 27 us; skew 500 ppm
Oct 26 15:07:53 dist-test-slave-8pgz krb5kdc[18185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491273, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:53.854555 18201 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:07:53.855103 18201 webserver.cc:492] Webserver started at http://127.14.212.126:45095/ using document root <none> and password file <none>
I20251026 15:07:53.855288 18201 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:53.855327 18201 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:53.855439 18201 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:53.856410 18201 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "cda8417451e245f289b3f39056075640"
format_stamp: "Formatted at 2025-10-26 15:07:53 on dist-test-slave-8pgz"
server_key: "a4614684806d264b39ba37fe4befa23f"
server_key_iv: "e1c0633d0da8ecbbdbad2cf7cd7ed6aa"
server_key_version: "encryptionkey@0"
I20251026 15:07:53.856721 18201 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "cda8417451e245f289b3f39056075640"
format_stamp: "Formatted at 2025-10-26 15:07:53 on dist-test-slave-8pgz"
server_key: "a4614684806d264b39ba37fe4befa23f"
server_key_iv: "e1c0633d0da8ecbbdbad2cf7cd7ed6aa"
server_key_version: "encryptionkey@0"
I20251026 15:07:53.857961 18201 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:07:53.858673 18217 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:53.858847 18201 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:53.858925 18201 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "cda8417451e245f289b3f39056075640"
format_stamp: "Formatted at 2025-10-26 15:07:53 on dist-test-slave-8pgz"
server_key: "a4614684806d264b39ba37fe4befa23f"
server_key_iv: "e1c0633d0da8ecbbdbad2cf7cd7ed6aa"
server_key_version: "encryptionkey@0"
I20251026 15:07:53.858985 18201 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:53.869609 18201 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:53.875909 18201 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:53.876071 18201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:53.880020 18201 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:41479
I20251026 15:07:53.880064 18270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:41479 every 8 connection(s)
I20251026 15:07:53.880373 18201 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:07:53.880977 18271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:53.883497 18271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640: Bootstrap starting.
I20251026 15:07:53.884078 18271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:53.884326 18271 log.cc:826] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:53.885092 18271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640: No bootstrap required, opened a new log
I20251026 15:07:53.886251 18271 raft_consensus.cc:359] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda8417451e245f289b3f39056075640" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41479 } }
I20251026 15:07:53.886368 18271 raft_consensus.cc:385] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:53.886386 18271 raft_consensus.cc:740] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda8417451e245f289b3f39056075640, State: Initialized, Role: FOLLOWER
I20251026 15:07:53.886548 18271 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda8417451e245f289b3f39056075640" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41479 } }
I20251026 15:07:53.886620 18271 raft_consensus.cc:399] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:07:53.886652 18271 raft_consensus.cc:493] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:07:53.886629 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 18201
I20251026 15:07:53.886696 18271 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:53.886741 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:07:53.886906 15185 external_mini_cluster.cc:1468] Setting key 8e4b6caeaa470c6113901dd461c58815
I20251026 15:07:53.887274 18271 raft_consensus.cc:515] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda8417451e245f289b3f39056075640" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41479 } }
I20251026 15:07:53.887379 18271 leader_election.cc:304] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda8417451e245f289b3f39056075640; no voters:
I20251026 15:07:53.887542 18271 leader_election.cc:290] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:07:53.887657 18276 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:53.887751 18271 sys_catalog.cc:565] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:07:53.887842 18276 raft_consensus.cc:697] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [term 1 LEADER]: Becoming Leader. State: Replica: cda8417451e245f289b3f39056075640, State: Running, Role: LEADER
I20251026 15:07:53.887939 18276 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda8417451e245f289b3f39056075640" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41479 } }
I20251026 15:07:53.888343 18277 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cda8417451e245f289b3f39056075640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda8417451e245f289b3f39056075640" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41479 } } }
I20251026 15:07:53.888437 18277 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [sys.catalog]: This master's current role is: LEADER
I20251026 15:07:53.888666 18278 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda8417451e245f289b3f39056075640. Latest consensus state: current_term: 1 leader_uuid: "cda8417451e245f289b3f39056075640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda8417451e245f289b3f39056075640" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41479 } } }
I20251026 15:07:53.889003 18278 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640 [sys.catalog]: This master's current role is: LEADER
W20251026 15:07:53.889681 18292 catalog_manager.cc:1568] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:07:53.889724 18292 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:07:53.889767 18285 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:07:53.889894 18285 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:07:53.891149 18285 catalog_manager.cc:1357] Generated new cluster ID: ae7b57b78a1b4f019b3433e847b17e9b
I20251026 15:07:53.891198 18285 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 26 15:07:53 dist-test-slave-8pgz krb5kdc[18185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:53.895875 18285 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:07:53.896564 18285 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:07:53.900579 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:53.887437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.14.212.126:41479)
1026 15:07:53.887602 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:53.887619 (+ 17us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:53.887643 (+ 24us) server_negotiation.cc:408] Connection header received
1026 15:07:53.887759 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:53.887764 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:53.887820 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:53.888829 (+ 1009us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:53.889531 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:53.891044 (+ 1513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:53.891860 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:53.892487 (+ 627us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:53.894679 (+ 2192us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:53.894685 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:53.894685 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:53.894706 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:53.899264 (+ 4558us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:53.899584 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:53.899587 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:53.899587 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:53.899639 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:53.899781 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:53.899782 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:53.899783 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:53.899986 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:53.900128 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:53.900274 (+ 146us) server_negotiation.cc:300] Negotiation successful
1026 15:07:53.900416 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20251026 15:07:53.905200 18285 catalog_manager.cc:6022] T 00000000000000000000000000000000 P cda8417451e245f289b3f39056075640: Generated new TSK 0
I20251026 15:07:53.905381 18285 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:53.962255 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41479
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:54.044356 18299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:54.044562 18299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:54.044600 18299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:54.046057 18299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:54.046108 18299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:54.046150 18299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:07:54.047838 18299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:41479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:54.048095 18299 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:54.048280 18299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:54.051152 18307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:54.051172 18304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:54.051153 18305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:54.051322 18299 server_base.cc:1047] running on GCE node
I20251026 15:07:54.051553 18299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:54.051787 18299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:54.052954 18299 hybrid_clock.cc:648] HybridClock initialized: now 1761491274052932 us; error 43 us; skew 500 ppm
Oct 26 15:07:54 dist-test-slave-8pgz krb5kdc[18185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491274, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:54.055629 18299 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:07:54.056174 18299 webserver.cc:492] Webserver started at http://127.14.212.65:43247/ using document root <none> and password file <none>
I20251026 15:07:54.056368 18299 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:54.056408 18299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:54.056524 18299 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:54.057448 18299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "9d580cafe0e14426833a26f69f0230d7"
format_stamp: "Formatted at 2025-10-26 15:07:54 on dist-test-slave-8pgz"
server_key: "5a2875ba1880e49d5a8a96c205ddf4d8"
server_key_iv: "89fc638769ab996a734a968fa5282733"
server_key_version: "encryptionkey@0"
I20251026 15:07:54.057765 18299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "9d580cafe0e14426833a26f69f0230d7"
format_stamp: "Formatted at 2025-10-26 15:07:54 on dist-test-slave-8pgz"
server_key: "5a2875ba1880e49d5a8a96c205ddf4d8"
server_key_iv: "89fc638769ab996a734a968fa5282733"
server_key_version: "encryptionkey@0"
I20251026 15:07:54.059053 18299 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:07:54.059742 18314 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:54.059933 18299 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:07:54.060034 18299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "9d580cafe0e14426833a26f69f0230d7"
format_stamp: "Formatted at 2025-10-26 15:07:54 on dist-test-slave-8pgz"
server_key: "5a2875ba1880e49d5a8a96c205ddf4d8"
server_key_iv: "89fc638769ab996a734a968fa5282733"
server_key_version: "encryptionkey@0"
I20251026 15:07:54.060096 18299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:54.077322 18299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:54.079507 18299 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:54.079655 18299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:54.079887 18299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:07:54.080197 18299 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:54.080231 18299 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:54.080269 18299 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:54.080299 18299 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:54.086043 18299 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:36675
I20251026 15:07:54.086145 18428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:36675 every 8 connection(s)
I20251026 15:07:54.086433 18299 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:07:54.087231 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 18299
I20251026 15:07:54.087342 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:07:54.087536 15185 external_mini_cluster.cc:1468] Setting key 70025f9032aaceb770a0bce82ff7def2
Oct 26 15:07:54 dist-test-slave-8pgz krb5kdc[18185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491274, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
I20251026 15:07:54.096397 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:54.087237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:46859 (local address 127.14.212.126:41479)
1026 15:07:54.087339 (+ 102us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:54.087342 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:54.087423 (+ 81us) server_negotiation.cc:408] Connection header received
1026 15:07:54.088107 (+ 684us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:54.088108 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:54.088122 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:54.088177 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:54.089299 (+ 1122us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:54.089987 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:54.090588 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:54.090837 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:54.093285 (+ 2448us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:54.093289 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:54.093290 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:54.093297 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:54.094932 (+ 1635us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:54.095418 (+ 486us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:54.095419 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:54.095419 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:54.095444 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:54.095661 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:54.095661 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:54.095661 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:54.095749 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:54.095797 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:54.096202 (+ 405us) server_negotiation.cc:300] Negotiation successful
1026 15:07:54.096295 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251026 15:07:54.096467 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:54.087139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:41479 (local address 127.14.212.65:46859)
1026 15:07:54.087296 (+ 157us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:54.087338 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:54.088008 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:54.088240 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:54.088243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:54.088756 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:54.089197 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:54.089202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:54.090089 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:54.090090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:54.090514 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:54.090516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:54.090630 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:54.091605 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:54.091622 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:54.093171 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:54.095022 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:54.095030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:54.095033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:54.095336 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:54.095515 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:54.095515 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:54.095516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:54.095604 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:54.095830 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:54.095834 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:54.095959 (+ 125us) client_negotiation.cc:769] Sending connection context
1026 15:07:54.096134 (+ 175us) client_negotiation.cc:240] Negotiation successful
1026 15:07:54.096283 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20251026 15:07:54.097124 18429 heartbeater.cc:344] Connected to a master server at 127.14.212.126:41479
I20251026 15:07:54.097213 18429 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:54.097466 18429 heartbeater.cc:507] Master 127.14.212.126:41479 requested a full tablet report, sending...
I20251026 15:07:54.097932 18234 ts_manager.cc:194] Registered new tserver with Master: 9d580cafe0e14426833a26f69f0230d7 (127.14.212.65:36675)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
I20251026 15:07:54.098649 18234 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:46859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:54.137389 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41479
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:54.219340 18436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:54.219539 18436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:54.219578 18436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:54.221025 18436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:54.221071 18436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:54.221113 18436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:07:54.222734 18436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:41479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:54.222986 18436 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:54.223182 18436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:54.226020 18441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:54.226035 18444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:54.226081 18436 server_base.cc:1047] running on GCE node
W20251026 15:07:54.226085 18442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:54.226445 18436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:54.226679 18436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:54.227842 18436 hybrid_clock.cc:648] HybridClock initialized: now 1761491274227821 us; error 42 us; skew 500 ppm
Oct 26 15:07:54 dist-test-slave-8pgz krb5kdc[18185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491274, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:54.230404 18436 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:07:54.231002 18436 webserver.cc:492] Webserver started at http://127.14.212.66:40035/ using document root <none> and password file <none>
I20251026 15:07:54.231185 18436 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:54.231225 18436 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:54.231343 18436 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:54.232317 18436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "302c175cecdc46e492cb5676c347c938"
format_stamp: "Formatted at 2025-10-26 15:07:54 on dist-test-slave-8pgz"
server_key: "7ff741b2bcd2e3ec940010098947941a"
server_key_iv: "61ba78ef623e9e5d5615b4a47aa7a84e"
server_key_version: "encryptionkey@0"
I20251026 15:07:54.232610 18436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "302c175cecdc46e492cb5676c347c938"
format_stamp: "Formatted at 2025-10-26 15:07:54 on dist-test-slave-8pgz"
server_key: "7ff741b2bcd2e3ec940010098947941a"
server_key_iv: "61ba78ef623e9e5d5615b4a47aa7a84e"
server_key_version: "encryptionkey@0"
I20251026 15:07:54.233891 18436 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:07:54.234721 18451 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:54.234907 18436 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:54.234992 18436 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "302c175cecdc46e492cb5676c347c938"
format_stamp: "Formatted at 2025-10-26 15:07:54 on dist-test-slave-8pgz"
server_key: "7ff741b2bcd2e3ec940010098947941a"
server_key_iv: "61ba78ef623e9e5d5615b4a47aa7a84e"
server_key_version: "encryptionkey@0"
I20251026 15:07:54.235075 18436 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:54.244401 18436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:54.246202 18436 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:54.246337 18436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:54.246579 18436 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:07:54.246959 18436 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:54.247020 18436 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:54.247057 18436 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:54.247083 18436 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:54.253244 18436 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:38989
I20251026 15:07:54.253305 18565 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:38989 every 8 connection(s)
I20251026 15:07:54.253607 18436 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
Oct 26 15:07:54 dist-test-slave-8pgz krb5kdc[18185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491274, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:54.262079 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 18436
I20251026 15:07:54.262185 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:07:54.262362 15185 external_mini_cluster.cc:1468] Setting key 55dd6b9896f8c9c6be2a3a23a36dbe30
I20251026 15:07:54.262585 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:54.254300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:37145 (local address 127.14.212.126:41479)
1026 15:07:54.254365 (+ 65us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:54.254368 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:54.254648 (+ 280us) server_negotiation.cc:408] Connection header received
1026 15:07:54.255063 (+ 415us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:54.255065 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:54.255075 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:54.255135 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:54.255914 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:54.256464 (+ 550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:54.257010 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:54.257143 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:54.259474 (+ 2331us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:54.259479 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:54.259480 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:54.259487 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:54.261009 (+ 1522us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:54.261419 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:54.261420 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:54.261420 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:54.261444 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:54.261749 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:54.261750 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:54.261750 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:54.261875 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:54.261959 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:54.262339 (+ 380us) server_negotiation.cc:300] Negotiation successful
1026 15:07:54.262491 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251026 15:07:54.262727 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:54.254264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:41479 (local address 127.14.212.66:37145)
1026 15:07:54.254445 (+ 181us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:54.254476 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:54.254961 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:54.255187 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:54.255191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:54.255498 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:54.255807 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:54.255810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:54.256542 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:54.256543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:54.256921 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:54.256923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:54.257024 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:54.257938 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:54.257952 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:54.259391 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:54.261111 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:54.261117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:54.261120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:54.261338 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:54.261585 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:54.261586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:54.261587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:54.261671 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:54.261975 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:54.261981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:54.262121 (+ 140us) client_negotiation.cc:769] Sending connection context
1026 15:07:54.262291 (+ 170us) client_negotiation.cc:240] Negotiation successful
1026 15:07:54.262493 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20251026 15:07:54.263346 18566 heartbeater.cc:344] Connected to a master server at 127.14.212.126:41479
I20251026 15:07:54.263473 18566 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:54.263723 18566 heartbeater.cc:507] Master 127.14.212.126:41479 requested a full tablet report, sending...
I20251026 15:07:54.264158 18234 ts_manager.cc:194] Registered new tserver with Master: 302c175cecdc46e492cb5676c347c938 (127.14.212.66:38989)
I20251026 15:07:54.264705 18234 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:37145
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:54.311290 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41479
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:54.393098 18573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:54.393296 18573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:54.393324 18573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:54.394785 18573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:54.394834 18573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:54.394862 18573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:07:54.396479 18573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:41479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:54.396708 18573 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:54.396944 18573 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:07:54.399868 18573 server_base.cc:1047] running on GCE node
W20251026 15:07:54.399788 18578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:54.399813 18581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:54.399838 18579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:54.400235 18573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:54.400436 18573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:54.401590 18573 hybrid_clock.cc:648] HybridClock initialized: now 1761491274401574 us; error 32 us; skew 500 ppm
Oct 26 15:07:54 dist-test-slave-8pgz krb5kdc[18185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491274, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:54.404382 18573 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:07:54.404959 18573 webserver.cc:492] Webserver started at http://127.14.212.67:35747/ using document root <none> and password file <none>
I20251026 15:07:54.405190 18573 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:54.405233 18573 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:54.405350 18573 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:54.406329 18573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "5687bda4928e45efad1970640cd4bec1"
format_stamp: "Formatted at 2025-10-26 15:07:54 on dist-test-slave-8pgz"
server_key: "adc3bee9c2f94751bcf48a2da421b2c3"
server_key_iv: "1c09530f6a34cec61b588af9113b61d3"
server_key_version: "encryptionkey@0"
I20251026 15:07:54.406683 18573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "5687bda4928e45efad1970640cd4bec1"
format_stamp: "Formatted at 2025-10-26 15:07:54 on dist-test-slave-8pgz"
server_key: "adc3bee9c2f94751bcf48a2da421b2c3"
server_key_iv: "1c09530f6a34cec61b588af9113b61d3"
server_key_version: "encryptionkey@0"
I20251026 15:07:54.408028 18573 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:54.408838 18588 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:54.409068 18573 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:07:54.409150 18573 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "5687bda4928e45efad1970640cd4bec1"
format_stamp: "Formatted at 2025-10-26 15:07:54 on dist-test-slave-8pgz"
server_key: "adc3bee9c2f94751bcf48a2da421b2c3"
server_key_iv: "1c09530f6a34cec61b588af9113b61d3"
server_key_version: "encryptionkey@0"
I20251026 15:07:54.409219 18573 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:54.425345 18573 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:54.427512 18573 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:54.427672 18573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:54.427910 18573 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:07:54.428237 18573 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:07:54.428270 18573 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:54.428308 18573 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:07:54.428341 18573 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:54.434091 18573 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:41679
I20251026 15:07:54.434152 18703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:41679 every 8 connection(s)
I20251026 15:07:54.434501 18573 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:07:54.436118 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 18573
I20251026 15:07:54.436214 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:07:54.436352 15185 external_mini_cluster.cc:1468] Setting key 87e994c3e8d36d7b96dea0078e0b98e9
Oct 26 15:07:54 dist-test-slave-8pgz krb5kdc[18185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491274, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:54.443347 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:54.435210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:49037 (local address 127.14.212.126:41479)
1026 15:07:54.435272 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:54.435274 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:54.435445 (+ 171us) server_negotiation.cc:408] Connection header received
1026 15:07:54.435976 (+ 531us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:54.435977 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:54.435991 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:54.436060 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:54.436937 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:54.437450 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:54.438013 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:54.438182 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:54.440703 (+ 2521us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:54.440708 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:54.440709 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:54.440716 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:54.442070 (+ 1354us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:54.442492 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:54.442494 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:54.442494 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:54.442522 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:54.442700 (+ 178us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:54.442700 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:54.442701 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:54.442799 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:54.442867 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:54.443154 (+ 287us) server_negotiation.cc:300] Negotiation successful
1026 15:07:54.443255 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251026 15:07:54.443499 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:54.435153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:41479 (local address 127.14.212.67:49037)
1026 15:07:54.435332 (+ 179us) negotiation.cc:107] Waiting for socket to connect
1026 15:07:54.435359 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:07:54.435867 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:07:54.436131 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:07:54.436135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:07:54.436507 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:07:54.436851 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:54.436854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:54.437536 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:07:54.437537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:07:54.437938 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:07:54.437940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:07:54.438075 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:54.439146 (+ 1071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:07:54.439164 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:07:54.440596 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:07:54.442172 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:54.442179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:54.442182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:54.442424 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:54.442578 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:07:54.442578 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:07:54.442579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:07:54.442646 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:07:54.442861 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:07:54.442866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:07:54.443003 (+ 137us) client_negotiation.cc:769] Sending connection context
1026 15:07:54.443124 (+ 121us) client_negotiation.cc:240] Negotiation successful
1026 15:07:54.443275 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
I20251026 15:07:54.444025 18704 heartbeater.cc:344] Connected to a master server at 127.14.212.126:41479
I20251026 15:07:54.444123 18704 heartbeater.cc:461] Registering TS with master...
I20251026 15:07:54.444307 18704 heartbeater.cc:507] Master 127.14.212.126:41479 requested a full tablet report, sending...
I20251026 15:07:54.444689 18234 ts_manager.cc:194] Registered new tserver with Master: 5687bda4928e45efad1970640cd4bec1 (127.14.212.67:41679)
I20251026 15:07:54.445079 18234 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:49037
I20251026 15:07:54.448238 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251026 15:07:54.546017 18275 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:07:54.544317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.14.212.126:41479)
1026 15:07:54.544387 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:54.544390 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:54.544583 (+ 193us) server_negotiation.cc:408] Connection header received
1026 15:07:54.545017 (+ 434us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:54.545018 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:54.545028 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:54.545094 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:54.545955 (+ 861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56956: BlockingRecv error: recv got EOF from 127.0.0.1:56956 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20251026 15:07:54.546689 18275 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:07:54.546230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.14.212.126:41479)
1026 15:07:54.546280 (+ 50us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:54.546282 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:54.546294 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:07:54.546317 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:54.546318 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:54.546328 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:54.546413 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:54.546642 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56966: BlockingRecv error: recv got EOF from 127.0.0.1:56966 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20251026 15:07:54.547260 18275 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:07:54.546885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56968 (local address 127.14.212.126:41479)
1026 15:07:54.546918 (+ 33us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:54.546920 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:54.546928 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:07:54.546994 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:54.546994 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:54.547001 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:54.547046 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:54.547229 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56968: BlockingRecv error: recv got EOF from 127.0.0.1:56968 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20251026 15:07:54.548849 18275 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:07:54.548438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.14.212.126:41479)
1026 15:07:54.548482 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:54.548483 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:54.548490 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:07:54.548536 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:54.548537 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:54.548544 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:54.548591 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:54.548810 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56978: BlockingRecv error: recv got EOF from 127.0.0.1:56978 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20251026 15:07:54.554373 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 18299
I20251026 15:07:54.559353 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 18436
I20251026 15:07:54.563910 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 18573
I20251026 15:07:54.568305 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 18201
2025-10-26T15:07:54Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (2948 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:07:54 dist-test-slave-8pgz krb5kdc[18727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:07:54 dist-test-slave-8pgz krb5kdc[18727](info): set up 2 sockets
Oct 26 15:07:54 dist-test-slave-8pgz krb5kdc[18727](info): commencing operation
krb5kdc: starting...
W20251026 15:07:56.555871 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.967s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:07:56 dist-test-slave-8pgz krb5kdc[18727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:07:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:07:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:07:56.691179 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:41413
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:41413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:07:56.771032 18743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:07:56.771217 18743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:07:56.771241 18743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:07:56.772650 18743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:07:56.772694 18743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:07:56.772709 18743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:07:56.772722 18743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:07:56.772733 18743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:07:56.774263 18743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:41413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:41413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:07:56.774525 18743 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:07:56.774691 18743 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:07:56.777204 18748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:56.777205 18749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:07:56.777199 18751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:07:56.777271 18743 server_base.cc:1047] running on GCE node
I20251026 15:07:56.777655 18743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:07:56.777913 18743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:07:56.779049 18743 hybrid_clock.cc:648] HybridClock initialized: now 1761491276779024 us; error 40 us; skew 500 ppm
Oct 26 15:07:56 dist-test-slave-8pgz krb5kdc[18727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491276, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:07:56.781317 18743 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:07:56.781831 18743 webserver.cc:492] Webserver started at http://127.14.212.126:38755/ using document root <none> and password file <none>
I20251026 15:07:56.782022 18743 fs_manager.cc:362] Metadata directory not provided
I20251026 15:07:56.782059 18743 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:07:56.782166 18743 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:07:56.783193 18743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "2c925bb9dbde4847a2d00ae64bf426c9"
format_stamp: "Formatted at 2025-10-26 15:07:56 on dist-test-slave-8pgz"
server_key: "6d602c254ae2e50849b269b205931bb4"
server_key_iv: "bdab5d98b7d0656acd87606bb0ffc177"
server_key_version: "encryptionkey@0"
I20251026 15:07:56.783516 18743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "2c925bb9dbde4847a2d00ae64bf426c9"
format_stamp: "Formatted at 2025-10-26 15:07:56 on dist-test-slave-8pgz"
server_key: "6d602c254ae2e50849b269b205931bb4"
server_key_iv: "bdab5d98b7d0656acd87606bb0ffc177"
server_key_version: "encryptionkey@0"
I20251026 15:07:56.784698 18743 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:07:56.785351 18759 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:07:56.785511 18743 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:07:56.785579 18743 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "2c925bb9dbde4847a2d00ae64bf426c9"
format_stamp: "Formatted at 2025-10-26 15:07:56 on dist-test-slave-8pgz"
server_key: "6d602c254ae2e50849b269b205931bb4"
server_key_iv: "bdab5d98b7d0656acd87606bb0ffc177"
server_key_version: "encryptionkey@0"
I20251026 15:07:56.785657 18743 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:07:56.806275 18743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:07:56.808718 18743 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:07:56.808861 18743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:07:56.812820 18743 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:41413
I20251026 15:07:56.812860 18811 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:41413 every 8 connection(s)
I20251026 15:07:56.813179 18743 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:07:56.813787 18812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:07:56.816283 18812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9: Bootstrap starting.
I20251026 15:07:56.816671 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 18743
I20251026 15:07:56.816751 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:07:56.816912 15185 external_mini_cluster.cc:1468] Setting key 474a060f60c8cf22639843982fb9319e
I20251026 15:07:56.816912 18812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9: Neither blocks nor log segments found. Creating new log.
I20251026 15:07:56.817147 18812 log.cc:826] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9: Log is configured to *not* fsync() on all Append() calls
I20251026 15:07:56.817873 18812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9: No bootstrap required, opened a new log
I20251026 15:07:56.819382 18812 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c925bb9dbde4847a2d00ae64bf426c9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41413 } }
I20251026 15:07:56.819504 18812 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:07:56.819525 18812 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c925bb9dbde4847a2d00ae64bf426c9, State: Initialized, Role: FOLLOWER
I20251026 15:07:56.819614 18812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c925bb9dbde4847a2d00ae64bf426c9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41413 } }
I20251026 15:07:56.819675 18812 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:07:56.819710 18812 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:07:56.819754 18812 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:07:56.820379 18812 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c925bb9dbde4847a2d00ae64bf426c9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41413 } }
I20251026 15:07:56.820492 18812 leader_election.cc:304] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c925bb9dbde4847a2d00ae64bf426c9; no voters:
I20251026 15:07:56.820653 18812 leader_election.cc:290] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:07:56.820732 18817 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:07:56.820881 18812 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:07:56.820915 18817 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [term 1 LEADER]: Becoming Leader. State: Replica: 2c925bb9dbde4847a2d00ae64bf426c9, State: Running, Role: LEADER
I20251026 15:07:56.821020 18817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c925bb9dbde4847a2d00ae64bf426c9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41413 } }
I20251026 15:07:56.821293 18817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c925bb9dbde4847a2d00ae64bf426c9. Latest consensus state: current_term: 1 leader_uuid: "2c925bb9dbde4847a2d00ae64bf426c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c925bb9dbde4847a2d00ae64bf426c9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41413 } } }
I20251026 15:07:56.821359 18817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:07:56.821525 18821 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:07:56.821945 18821 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Oct 26 15:07:56 dist-test-slave-8pgz krb5kdc[18727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:07:56.821327 18818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c925bb9dbde4847a2d00ae64bf426c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c925bb9dbde4847a2d00ae64bf426c9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41413 } } }
I20251026 15:07:56.823813 18818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:07:56.824153 18821 catalog_manager.cc:1357] Generated new cluster ID: 4d413ae24eef44298e6458f3204adb78
I20251026 15:07:56.824205 18821 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:07:56.827538 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:56.817394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.14.212.126:41413)
1026 15:07:56.817629 (+ 235us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:56.817646 (+ 17us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:56.817665 (+ 19us) server_negotiation.cc:408] Connection header received
1026 15:07:56.818033 (+ 368us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:56.818039 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:56.818088 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:56.818197 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:56.819164 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:56.820057 (+ 893us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:56.820814 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:56.821083 (+ 269us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:56.823783 (+ 2700us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:56.823791 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:56.823791 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:07:56.823996 (+ 205us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:56.826216 (+ 2220us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:56.826599 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:56.826602 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:56.826602 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:56.826660 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:07:56.826820 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:07:56.826821 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:07:56.826821 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:07:56.827020 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:56.827156 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:56.827278 (+ 122us) server_negotiation.cc:300] Negotiation successful
1026 15:07:56.827404 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20251026 15:07:56.828328 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:07:56.838263 18821 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:07:56.839131 18821 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:07:56.845150 18821 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 2c925bb9dbde4847a2d00ae64bf426c9: Generated new TSK 0
I20251026 15:07:56.845341 18821 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:07:56.851225 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:07:56.847906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.14.212.126:41413)
1026 15:07:56.847953 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1026 15:07:56.847955 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:07:56.848043 (+ 88us) server_negotiation.cc:408] Connection header received
1026 15:07:56.848114 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:07:56.848115 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:07:56.848124 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:07:56.848178 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:07:56.848791 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:56.849285 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:07:56.850066 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:07:56.850175 (+ 109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:07:56.850742 (+ 567us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:07:56.850745 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:07:56.850745 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:07:56.850953 (+ 208us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:07:56.850967 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:07:56.850972 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:07:56.851021 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1026 15:07:56.851111 (+ 90us) server_negotiation.cc:300] Negotiation successful
1026 15:07:56.851137 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251026 15:07:56.853312 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 18743
2025-10-26T15:07:56Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2283 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:07:58 dist-test-slave-8pgz krb5kdc[18845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:07:58 dist-test-slave-8pgz krb5kdc[18845](info): set up 2 sockets
Oct 26 15:07:58 dist-test-slave-8pgz krb5kdc[18845](info): commencing operation
krb5kdc: starting...
W20251026 15:08:00.559501 15185 mini_kdc.cc:121] Time spent starting KDC: real 3.688s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:08:00 dist-test-slave-8pgz krb5kdc[18845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:08:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:00.698973 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:35083
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:35083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:00.779960 18861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:00.780162 18861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:00.780187 18861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:00.781615 18861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:00.781654 18861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:00.781668 18861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:00.781682 18861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:00.781692 18861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:08:00.783370 18861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:35083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:35083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:00.783577 18861 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:00.783733 18861 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:00.786556 18867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:00.786573 18861 server_base.cc:1047] running on GCE node
W20251026 15:08:00.786556 18866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:00.786551 18869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:00.787057 18861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:00.787339 18861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:00.788487 18861 hybrid_clock.cc:648] HybridClock initialized: now 1761491280788469 us; error 32 us; skew 500 ppm
Oct 26 15:08:00 dist-test-slave-8pgz krb5kdc[18845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491280, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:00.791206 18861 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:08:00.791831 18861 webserver.cc:492] Webserver started at http://127.14.212.126:46621/ using document root <none> and password file <none>
I20251026 15:08:00.792021 18861 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:00.792070 18861 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:00.792191 18861 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:00.793196 18861 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "b058919aec4a45c7a4cc0c22169fb624"
format_stamp: "Formatted at 2025-10-26 15:08:00 on dist-test-slave-8pgz"
server_key: "83b3c10dd1138c990e7f619c78123d3b"
server_key_iv: "8e2fa40c52c19143df44da3cabe192c5"
server_key_version: "encryptionkey@0"
I20251026 15:08:00.793524 18861 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "b058919aec4a45c7a4cc0c22169fb624"
format_stamp: "Formatted at 2025-10-26 15:08:00 on dist-test-slave-8pgz"
server_key: "83b3c10dd1138c990e7f619c78123d3b"
server_key_iv: "8e2fa40c52c19143df44da3cabe192c5"
server_key_version: "encryptionkey@0"
I20251026 15:08:00.794888 18861 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:08:00.795655 18876 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:00.795855 18861 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:00.795938 18861 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "b058919aec4a45c7a4cc0c22169fb624"
format_stamp: "Formatted at 2025-10-26 15:08:00 on dist-test-slave-8pgz"
server_key: "83b3c10dd1138c990e7f619c78123d3b"
server_key_iv: "8e2fa40c52c19143df44da3cabe192c5"
server_key_version: "encryptionkey@0"
I20251026 15:08:00.796005 18861 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:00.806257 18861 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:00.813879 18861 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:00.814043 18861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:00.818437 18861 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:35083
I20251026 15:08:00.818501 18928 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:35083 every 8 connection(s)
I20251026 15:08:00.818877 18861 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:00.819533 18929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:00.822209 18929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624: Bootstrap starting.
I20251026 15:08:00.822844 18929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:00.823081 18929 log.cc:826] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:00.823693 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 18861
I20251026 15:08:00.823804 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:08:00.824013 18929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624: No bootstrap required, opened a new log
I20251026 15:08:00.824021 15185 external_mini_cluster.cc:1468] Setting key a999eb27fb39a6b324554bb652381711
I20251026 15:08:00.825505 18929 raft_consensus.cc:359] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b058919aec4a45c7a4cc0c22169fb624" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 35083 } }
I20251026 15:08:00.825644 18929 raft_consensus.cc:385] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:00.825677 18929 raft_consensus.cc:740] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b058919aec4a45c7a4cc0c22169fb624, State: Initialized, Role: FOLLOWER
I20251026 15:08:00.825784 18929 consensus_queue.cc:260] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b058919aec4a45c7a4cc0c22169fb624" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 35083 } }
I20251026 15:08:00.825848 18929 raft_consensus.cc:399] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:00.825886 18929 raft_consensus.cc:493] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:00.825942 18929 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:00.826614 18929 raft_consensus.cc:515] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b058919aec4a45c7a4cc0c22169fb624" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 35083 } }
I20251026 15:08:00.826735 18929 leader_election.cc:304] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b058919aec4a45c7a4cc0c22169fb624; no voters:
I20251026 15:08:00.826901 18929 leader_election.cc:290] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:08:00.826952 18935 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:00.827090 18935 raft_consensus.cc:697] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [term 1 LEADER]: Becoming Leader. State: Replica: b058919aec4a45c7a4cc0c22169fb624, State: Running, Role: LEADER
I20251026 15:08:00.827170 18929 sys_catalog.cc:565] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:00.827239 18935 consensus_queue.cc:237] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b058919aec4a45c7a4cc0c22169fb624" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 35083 } }
I20251026 15:08:00.827641 18935 sys_catalog.cc:455] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b058919aec4a45c7a4cc0c22169fb624. Latest consensus state: current_term: 1 leader_uuid: "b058919aec4a45c7a4cc0c22169fb624" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b058919aec4a45c7a4cc0c22169fb624" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 35083 } } }
I20251026 15:08:00.827649 18937 sys_catalog.cc:455] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b058919aec4a45c7a4cc0c22169fb624" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b058919aec4a45c7a4cc0c22169fb624" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 35083 } } }
I20251026 15:08:00.827890 18937 sys_catalog.cc:458] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:00.827797 18935 sys_catalog.cc:458] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:00.828794 18943 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:00.829321 18943 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Oct 26 15:08:00 dist-test-slave-8pgz krb5kdc[18845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:00.831137 18943 catalog_manager.cc:1357] Generated new cluster ID: 46be030b4b7542f0927167241bff9bc1
I20251026 15:08:00.831187 18943 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:08:00.835289 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:00.824554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.14.212.126:35083)
1026 15:08:00.824737 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:00.824747 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:00.824767 (+ 20us) server_negotiation.cc:408] Connection header received
1026 15:08:00.825477 (+ 710us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:00.825483 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:00.825532 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:00.825662 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:00.826621 (+ 959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:00.827701 (+ 1080us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:00.828741 (+ 1040us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:00.829048 (+ 307us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:00.831489 (+ 2441us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:00.831513 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:00.831513 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:00.831534 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:00.833615 (+ 2081us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:00.834051 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:00.834054 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:00.834055 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:00.834124 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:00.834301 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:00.834301 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:00.834301 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:00.834572 (+ 271us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:00.834700 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:00.834859 (+ 159us) server_negotiation.cc:300] Negotiation successful
1026 15:08:00.835093 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20251026 15:08:00.851776 18943 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:08:00.852591 18943 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:00.865592 18943 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b058919aec4a45c7a4cc0c22169fb624: Generated new TSK 0
I20251026 15:08:00.865864 18943 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:00.922725 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:35083
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:01.007714 18958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:01.007932 18958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:01.007958 18958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:01.009516 18958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:01.009577 18958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:01.009608 18958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:08:01.011435 18958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:01.011677 18958 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:01.011891 18958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:01.014916 18966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:01.015122 18958 server_base.cc:1047] running on GCE node
W20251026 15:08:01.014989 18963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:01.015033 18964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:01.015386 18958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:01.015590 18958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:01.016808 18958 hybrid_clock.cc:648] HybridClock initialized: now 1761491281016801 us; error 57 us; skew 500 ppm
Oct 26 15:08:01 dist-test-slave-8pgz krb5kdc[18845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:01.019871 18958 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:08:01.020424 18958 webserver.cc:492] Webserver started at http://127.14.212.65:35783/ using document root <none> and password file <none>
I20251026 15:08:01.020603 18958 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:01.020639 18958 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:01.020726 18958 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:01.021703 18958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "d301701c4d3046afa0022556f4f27521"
format_stamp: "Formatted at 2025-10-26 15:08:01 on dist-test-slave-8pgz"
server_key: "a417e1a5d461cb3dce1c9dbe923af894"
server_key_iv: "789348d726d1290dcb94b3a824ccad38"
server_key_version: "encryptionkey@0"
I20251026 15:08:01.022025 18958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "d301701c4d3046afa0022556f4f27521"
format_stamp: "Formatted at 2025-10-26 15:08:01 on dist-test-slave-8pgz"
server_key: "a417e1a5d461cb3dce1c9dbe923af894"
server_key_iv: "789348d726d1290dcb94b3a824ccad38"
server_key_version: "encryptionkey@0"
I20251026 15:08:01.023430 18958 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:08:01.024225 18973 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:01.024412 18958 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:01.024482 18958 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "d301701c4d3046afa0022556f4f27521"
format_stamp: "Formatted at 2025-10-26 15:08:01 on dist-test-slave-8pgz"
server_key: "a417e1a5d461cb3dce1c9dbe923af894"
server_key_iv: "789348d726d1290dcb94b3a824ccad38"
server_key_version: "encryptionkey@0"
I20251026 15:08:01.024536 18958 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:01.047940 18958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:01.050199 18958 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:01.050312 18958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:01.050544 18958 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:01.050889 18958 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:01.050922 18958 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:01.050947 18958 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:01.050959 18958 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:01.056667 18958 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:46681
I20251026 15:08:01.056777 19086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:46681 every 8 connection(s)
I20251026 15:08:01.057080 18958 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:08:01.058017 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 18958
I20251026 15:08:01.058171 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:08:01.058341 15185 external_mini_cluster.cc:1468] Setting key 8e3dcb8ffe4be117e436b794b810d2be
Oct 26 15:08:01 dist-test-slave-8pgz krb5kdc[18845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
I20251026 15:08:01.067931 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:01.057861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:47157 (local address 127.14.212.126:35083)
1026 15:08:01.057920 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:01.057922 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:01.058046 (+ 124us) server_negotiation.cc:408] Connection header received
1026 15:08:01.058666 (+ 620us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:01.058668 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:01.058681 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:01.058733 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:01.059573 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.060089 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:01.060656 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.060837 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:01.064146 (+ 3309us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:01.064151 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:01.064151 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:01.064160 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:01.066301 (+ 2141us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:01.066820 (+ 519us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:01.066822 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:01.066823 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:01.066858 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:01.067171 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:01.067172 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:01.067173 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:01.067328 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:01.067397 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:01.067750 (+ 353us) server_negotiation.cc:300] Negotiation successful
1026 15:08:01.067859 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20251026 15:08:01.068022 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:01.057743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:35083 (local address 127.14.212.65:47157)
1026 15:08:01.057921 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:01.057953 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:01.058559 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:01.058802 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:01.058807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:01.059162 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:01.059465 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:01.059468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.060182 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:01.060185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:01.060571 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:01.060573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.060714 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:01.061885 (+ 1171us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:01.061915 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:01.063955 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:01.066482 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:01.066492 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:01.066494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:01.066735 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:01.066964 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:01.066965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:01.066966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:01.067088 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:01.067419 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:01.067426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:01.067574 (+ 148us) client_negotiation.cc:769] Sending connection context
1026 15:08:01.067687 (+ 113us) client_negotiation.cc:240] Negotiation successful
1026 15:08:01.067830 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20251026 15:08:01.068727 19087 heartbeater.cc:344] Connected to a master server at 127.14.212.126:35083
I20251026 15:08:01.068820 19087 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:01.069022 19087 heartbeater.cc:507] Master 127.14.212.126:35083 requested a full tablet report, sending...
I20251026 15:08:01.069531 18893 ts_manager.cc:194] Registered new tserver with Master: d301701c4d3046afa0022556f4f27521 (127.14.212.65:46681)
I20251026 15:08:01.070204 18893 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:47157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:01.111655 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:35083
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:01.192673 19094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:01.192866 19094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:01.192898 19094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:01.194366 19094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:01.194439 19094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:01.194470 19094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:08:01.196161 19094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:01.196377 19094 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:01.196576 19094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:01.199352 19099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:01.199350 19100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:01.199344 19102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:01.199594 19094 server_base.cc:1047] running on GCE node
I20251026 15:08:01.199762 19094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:01.199973 19094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:01.201120 19094 hybrid_clock.cc:648] HybridClock initialized: now 1761491281201103 us; error 42 us; skew 500 ppm
Oct 26 15:08:01 dist-test-slave-8pgz krb5kdc[18845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:01.203768 19094 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:08:01.204286 19094 webserver.cc:492] Webserver started at http://127.14.212.66:35409/ using document root <none> and password file <none>
I20251026 15:08:01.204455 19094 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:01.204488 19094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:01.204569 19094 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:01.205478 19094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "060bf1bcc0084a4ea3e584f79fc5ed00"
format_stamp: "Formatted at 2025-10-26 15:08:01 on dist-test-slave-8pgz"
server_key: "a5eba60194aeec99e05d7978acc29c25"
server_key_iv: "c411d9bb20e540279e0afac34493677e"
server_key_version: "encryptionkey@0"
I20251026 15:08:01.205742 19094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "060bf1bcc0084a4ea3e584f79fc5ed00"
format_stamp: "Formatted at 2025-10-26 15:08:01 on dist-test-slave-8pgz"
server_key: "a5eba60194aeec99e05d7978acc29c25"
server_key_iv: "c411d9bb20e540279e0afac34493677e"
server_key_version: "encryptionkey@0"
I20251026 15:08:01.207059 19094 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:08:01.207696 19109 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:01.207924 19094 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:01.208002 19094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "060bf1bcc0084a4ea3e584f79fc5ed00"
format_stamp: "Formatted at 2025-10-26 15:08:01 on dist-test-slave-8pgz"
server_key: "a5eba60194aeec99e05d7978acc29c25"
server_key_iv: "c411d9bb20e540279e0afac34493677e"
server_key_version: "encryptionkey@0"
I20251026 15:08:01.208076 19094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:01.223330 19094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:01.225419 19094 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:01.225550 19094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:01.225787 19094 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:01.226094 19094 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:01.226132 19094 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:01.226157 19094 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:01.226184 19094 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:01.231778 19094 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:39773
I20251026 15:08:01.231861 19222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:39773 every 8 connection(s)
I20251026 15:08:01.232116 19094 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:08:01.236483 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 19094
I20251026 15:08:01.236585 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:08:01.236745 15185 external_mini_cluster.cc:1468] Setting key 8fc18c2bbe84c6b3ca77535286e8b60f
Oct 26 15:08:01 dist-test-slave-8pgz krb5kdc[18845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:01.240932 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:01.232761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:35083 (local address 127.14.212.66:37681)
1026 15:08:01.232941 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:01.232966 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:01.233443 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:01.233649 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:01.233652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:01.233971 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:01.234266 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:01.234270 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.234987 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:01.234989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:01.235381 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:01.235383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.235492 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:01.236396 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:01.236412 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:01.238007 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:01.239726 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:01.239733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:01.239735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:01.239935 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:01.240088 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:01.240089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:01.240090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:01.240155 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:01.240455 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:01.240458 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:01.240577 (+ 119us) client_negotiation.cc:769] Sending connection context
1026 15:08:01.240664 (+ 87us) client_negotiation.cc:240] Negotiation successful
1026 15:08:01.240770 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":68,"threads_started":1}
I20251026 15:08:01.240965 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:01.232858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:37681 (local address 127.14.212.126:35083)
1026 15:08:01.232936 (+ 78us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:01.232939 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:01.233047 (+ 108us) server_negotiation.cc:408] Connection header received
1026 15:08:01.233531 (+ 484us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:01.233533 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:01.233543 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:01.233595 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:01.234357 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.234891 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:01.235453 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.235559 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:01.238093 (+ 2534us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:01.238097 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:01.238097 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:01.238104 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:01.239640 (+ 1536us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:01.240006 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:01.240007 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:01.240008 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:01.240033 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:01.240239 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:01.240240 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:01.240241 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:01.240381 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:01.240471 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:01.240708 (+ 237us) server_negotiation.cc:300] Negotiation successful
1026 15:08:01.240822 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251026 15:08:01.241446 19223 heartbeater.cc:344] Connected to a master server at 127.14.212.126:35083
I20251026 15:08:01.241534 19223 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:01.241734 19223 heartbeater.cc:507] Master 127.14.212.126:35083 requested a full tablet report, sending...
I20251026 15:08:01.242125 18893 ts_manager.cc:194] Registered new tserver with Master: 060bf1bcc0084a4ea3e584f79fc5ed00 (127.14.212.66:39773)
I20251026 15:08:01.242547 18893 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:37681
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:01.289337 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:35083
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:01.369762 19230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:01.369972 19230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:01.369997 19230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:01.371539 19230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:01.371594 19230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:01.371623 19230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:08:01.373273 19230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:01.373461 19230 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:01.373617 19230 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:01.376365 19236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:01.376405 19238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:01.376430 19235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:01.376482 19230 server_base.cc:1047] running on GCE node
I20251026 15:08:01.376730 19230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:01.376964 19230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:01.378141 19230 hybrid_clock.cc:648] HybridClock initialized: now 1761491281378114 us; error 44 us; skew 500 ppm
Oct 26 15:08:01 dist-test-slave-8pgz krb5kdc[18845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:01.380713 19230 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:08:01.381271 19230 webserver.cc:492] Webserver started at http://127.14.212.67:42813/ using document root <none> and password file <none>
I20251026 15:08:01.381461 19230 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:01.381500 19230 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:01.381599 19230 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:01.382778 19230 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "807e4cca59164c68924bf5fcce81251d"
format_stamp: "Formatted at 2025-10-26 15:08:01 on dist-test-slave-8pgz"
server_key: "b64642947f50217830d56d0cda04be1b"
server_key_iv: "3f09aae4f014cc36b7fe6270d0034d2c"
server_key_version: "encryptionkey@0"
I20251026 15:08:01.383106 19230 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "807e4cca59164c68924bf5fcce81251d"
format_stamp: "Formatted at 2025-10-26 15:08:01 on dist-test-slave-8pgz"
server_key: "b64642947f50217830d56d0cda04be1b"
server_key_iv: "3f09aae4f014cc36b7fe6270d0034d2c"
server_key_version: "encryptionkey@0"
I20251026 15:08:01.384377 19230 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:01.385151 19245 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:01.385344 19230 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:01.385412 19230 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "807e4cca59164c68924bf5fcce81251d"
format_stamp: "Formatted at 2025-10-26 15:08:01 on dist-test-slave-8pgz"
server_key: "b64642947f50217830d56d0cda04be1b"
server_key_iv: "3f09aae4f014cc36b7fe6270d0034d2c"
server_key_version: "encryptionkey@0"
I20251026 15:08:01.385483 19230 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:01.395718 19230 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:01.397830 19230 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:01.397955 19230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:01.398195 19230 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:01.398559 19230 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:01.398595 19230 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:01.398619 19230 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:01.398648 19230 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:01.404417 19230 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:39621
I20251026 15:08:01.404503 19358 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:39621 every 8 connection(s)
I20251026 15:08:01.404793 19230 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
Oct 26 15:08:01 dist-test-slave-8pgz krb5kdc[18845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:01.413769 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:01.405519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:47497 (local address 127.14.212.126:35083)
1026 15:08:01.405604 (+ 85us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:01.405606 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:01.405711 (+ 105us) server_negotiation.cc:408] Connection header received
1026 15:08:01.406205 (+ 494us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:01.406207 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:01.406217 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:01.406286 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:01.407095 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.407583 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:01.408092 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.408198 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:01.410617 (+ 2419us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:01.410622 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:01.410622 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:01.410630 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:01.412266 (+ 1636us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:01.412654 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:01.412656 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:01.412656 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:01.412683 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:01.412887 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:01.412888 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:01.412889 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:01.413083 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:01.413173 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:01.413568 (+ 395us) server_negotiation.cc:300] Negotiation successful
1026 15:08:01.413669 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251026 15:08:01.413919 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:01.405455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:35083 (local address 127.14.212.67:47497)
1026 15:08:01.405601 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:01.405632 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:01.406109 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:01.406334 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:01.406337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:01.406711 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:01.407000 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:01.407003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.407650 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:01.407651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:01.408023 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:01.408024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.408122 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:01.409021 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:01.409036 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:01.410506 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:01.412358 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:01.412366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:01.412370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:01.412582 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:01.412739 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:01.412740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:01.412740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:01.412802 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:01.413172 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:01.413179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:01.413408 (+ 229us) client_negotiation.cc:769] Sending connection context
1026 15:08:01.413535 (+ 127us) client_negotiation.cc:240] Negotiation successful
1026 15:08:01.413680 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20251026 15:08:01.414067 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 19230
I20251026 15:08:01.414234 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:08:01.414479 15185 external_mini_cluster.cc:1468] Setting key 9c6c68be557a0b521aff4726f02e9431
I20251026 15:08:01.414454 19359 heartbeater.cc:344] Connected to a master server at 127.14.212.126:35083
I20251026 15:08:01.414561 19359 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:01.414789 19359 heartbeater.cc:507] Master 127.14.212.126:35083 requested a full tablet report, sending...
I20251026 15:08:01.415329 18892 ts_manager.cc:194] Registered new tserver with Master: 807e4cca59164c68924bf5fcce81251d (127.14.212.67:39621)
I20251026 15:08:01.415901 18892 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:47497
I20251026 15:08:01.426614 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 26 15:08:01 dist-test-slave-8pgz krb5kdc[18845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491281, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Oct 26 15:08:01 dist-test-slave-8pgz krb5kdc[18845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491281, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:01.448067 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:01.441250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37242 (local address 127.14.212.126:35083)
1026 15:08:01.441320 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:01.441322 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:01.441403 (+ 81us) server_negotiation.cc:408] Connection header received
1026 15:08:01.441472 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:01.441473 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:01.441482 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:01.441535 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:01.442294 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.442857 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:01.443436 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:01.443549 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:01.445616 (+ 2067us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:01.445620 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:01.445620 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:01.445627 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:01.447176 (+ 1549us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:01.447491 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:01.447492 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:01.447492 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:01.447517 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:01.447644 (+ 127us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:01.447644 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:01.447644 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:01.447759 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:01.447862 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:01.447913 (+ 51us) server_negotiation.cc:300] Negotiation successful
1026 15:08:01.447981 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20251026 15:08:01.448355 18892 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37242
I20251026 15:08:01.448796 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 18958
I20251026 15:08:01.453028 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 19094
I20251026 15:08:01.457834 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 19230
I20251026 15:08:01.462246 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 18861
2025-10-26T15:08:01Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4610 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:08:01 dist-test-slave-8pgz krb5kdc[19371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:08:01 dist-test-slave-8pgz krb5kdc[19371](info): set up 2 sockets
Oct 26 15:08:01 dist-test-slave-8pgz krb5kdc[19371](info): commencing operation
krb5kdc: starting...
W20251026 15:08:03.463891 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.981s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:08:03 dist-test-slave-8pgz krb5kdc[19371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:08:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:03.599078 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46509
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:46509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:03.680336 19388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:03.680544 19388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:03.680588 19388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:03.682077 19388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:03.682135 19388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:03.682157 19388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:03.682178 19388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:03.682196 19388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:08:03.683846 19388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:46509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:03.684137 19388 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:03.684325 19388 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:03.686797 19396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:03.686808 19394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:03.686873 19393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:03.686928 19388 server_base.cc:1047] running on GCE node
I20251026 15:08:03.687351 19388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:03.687556 19388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:03.688679 19388 hybrid_clock.cc:648] HybridClock initialized: now 1761491283688666 us; error 32 us; skew 500 ppm
Oct 26 15:08:03 dist-test-slave-8pgz krb5kdc[19371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491283, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:03.690946 19388 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:08:03.691447 19388 webserver.cc:492] Webserver started at http://127.14.212.126:40931/ using document root <none> and password file <none>
I20251026 15:08:03.691610 19388 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:03.691641 19388 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:03.691727 19388 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:03.692648 19388 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "722a831251a1492bb10069b59ba74e5c"
format_stamp: "Formatted at 2025-10-26 15:08:03 on dist-test-slave-8pgz"
server_key: "9e77a6e68176ca27319a93b010421770"
server_key_iv: "f4e902f244466f66355a9f979fdf2b8b"
server_key_version: "encryptionkey@0"
I20251026 15:08:03.692965 19388 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "722a831251a1492bb10069b59ba74e5c"
format_stamp: "Formatted at 2025-10-26 15:08:03 on dist-test-slave-8pgz"
server_key: "9e77a6e68176ca27319a93b010421770"
server_key_iv: "f4e902f244466f66355a9f979fdf2b8b"
server_key_version: "encryptionkey@0"
I20251026 15:08:03.694118 19388 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:08:03.694775 19403 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:03.694936 19388 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:03.695011 19388 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "722a831251a1492bb10069b59ba74e5c"
format_stamp: "Formatted at 2025-10-26 15:08:03 on dist-test-slave-8pgz"
server_key: "9e77a6e68176ca27319a93b010421770"
server_key_iv: "f4e902f244466f66355a9f979fdf2b8b"
server_key_version: "encryptionkey@0"
I20251026 15:08:03.695073 19388 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:03.711510 19388 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:03.713353 19388 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:03.713486 19388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:03.717406 19388 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:46509
I20251026 15:08:03.717456 19455 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:46509 every 8 connection(s)
I20251026 15:08:03.717779 19388 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:03.718454 19456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:03.720980 19456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c: Bootstrap starting.
I20251026 15:08:03.721572 19456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:03.721829 19456 log.cc:826] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:03.722594 19456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c: No bootstrap required, opened a new log
I20251026 15:08:03.723461 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 19388
I20251026 15:08:03.723563 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:08:03.723716 15185 external_mini_cluster.cc:1468] Setting key b45d8cccab5ce00d1bb0b99a3a683d5a
I20251026 15:08:03.723872 19456 raft_consensus.cc:359] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722a831251a1492bb10069b59ba74e5c" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46509 } }
I20251026 15:08:03.724009 19456 raft_consensus.cc:385] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:03.724040 19456 raft_consensus.cc:740] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722a831251a1492bb10069b59ba74e5c, State: Initialized, Role: FOLLOWER
I20251026 15:08:03.724148 19456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722a831251a1492bb10069b59ba74e5c" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46509 } }
I20251026 15:08:03.724227 19456 raft_consensus.cc:399] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:03.724291 19456 raft_consensus.cc:493] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:03.724356 19456 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:03.724905 19456 raft_consensus.cc:515] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722a831251a1492bb10069b59ba74e5c" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46509 } }
I20251026 15:08:03.725011 19456 leader_election.cc:304] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 722a831251a1492bb10069b59ba74e5c; no voters:
I20251026 15:08:03.725169 19456 leader_election.cc:290] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:08:03.725229 19461 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:03.725383 19461 raft_consensus.cc:697] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [term 1 LEADER]: Becoming Leader. State: Replica: 722a831251a1492bb10069b59ba74e5c, State: Running, Role: LEADER
I20251026 15:08:03.725405 19456 sys_catalog.cc:565] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:03.725533 19461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722a831251a1492bb10069b59ba74e5c" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46509 } }
I20251026 15:08:03.726665 19462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 722a831251a1492bb10069b59ba74e5c. Latest consensus state: current_term: 1 leader_uuid: "722a831251a1492bb10069b59ba74e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722a831251a1492bb10069b59ba74e5c" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46509 } } }
I20251026 15:08:03.726727 19463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "722a831251a1492bb10069b59ba74e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722a831251a1492bb10069b59ba74e5c" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46509 } } }
I20251026 15:08:03.726835 19462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:03.726892 19463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c [sys.catalog]: This master's current role is: LEADER
W20251026 15:08:03.727674 19477 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:08:03.727730 19477 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:08:03.727919 19478 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:03.728089 19478 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Oct 26 15:08:03 dist-test-slave-8pgz krb5kdc[19371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:03.729542 19478 catalog_manager.cc:1357] Generated new cluster ID: bd652a7725a14911ae2abcb01907175e
I20251026 15:08:03.729589 19478 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:08:03.733971 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:03.724242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.14.212.126:46509)
1026 15:08:03.724433 (+ 191us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:03.724445 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:03.724467 (+ 22us) server_negotiation.cc:408] Connection header received
1026 15:08:03.724564 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:03.724569 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:03.724607 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:03.724718 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:03.725648 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:03.726808 (+ 1160us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:03.727566 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:03.727823 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:03.729998 (+ 2175us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:03.730004 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:03.730004 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:03.730026 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:03.732089 (+ 2063us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:03.732485 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:03.732487 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:03.732488 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:03.732551 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:03.732715 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:03.732716 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:03.732716 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:03.732974 (+ 258us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:03.733138 (+ 164us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:03.733660 (+ 522us) server_negotiation.cc:300] Negotiation successful
1026 15:08:03.733792 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20251026 15:08:03.742267 19478 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:08:03.742817 19478 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:03.747090 19478 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 722a831251a1492bb10069b59ba74e5c: Generated new TSK 0
I20251026 15:08:03.747272 19478 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:03.801371 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46509
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:03.899955 19485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:03.900166 19485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:03.900204 19485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:03.901748 19485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:03.901810 19485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:03.901856 19485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:08:03.903620 19485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:03.903920 19485 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:03.904120 19485 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:03.906840 19490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:03.907153 19485 server_base.cc:1047] running on GCE node
W20251026 15:08:03.906840 19491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:03.906843 19493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:03.907503 19485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:03.907720 19485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:03.908859 19485 hybrid_clock.cc:648] HybridClock initialized: now 1761491283908846 us; error 32 us; skew 500 ppm
Oct 26 15:08:03 dist-test-slave-8pgz krb5kdc[19371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491283, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:03.911535 19485 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:08:03.912127 19485 webserver.cc:492] Webserver started at http://127.14.212.65:41173/ using document root <none> and password file <none>
I20251026 15:08:03.912343 19485 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:03.912380 19485 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:03.912494 19485 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:03.913465 19485 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "6edf9ea24b05406f8d65c4745abb7bd3"
format_stamp: "Formatted at 2025-10-26 15:08:03 on dist-test-slave-8pgz"
server_key: "7f4f543e4bd862bfb62525624fa4c5f7"
server_key_iv: "fc0ae84b8b08b78c92538c2a2db7a663"
server_key_version: "encryptionkey@0"
I20251026 15:08:03.913777 19485 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "6edf9ea24b05406f8d65c4745abb7bd3"
format_stamp: "Formatted at 2025-10-26 15:08:03 on dist-test-slave-8pgz"
server_key: "7f4f543e4bd862bfb62525624fa4c5f7"
server_key_iv: "fc0ae84b8b08b78c92538c2a2db7a663"
server_key_version: "encryptionkey@0"
I20251026 15:08:03.915081 19485 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:08:03.915778 19500 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:03.915947 19485 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:08:03.915999 19485 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "6edf9ea24b05406f8d65c4745abb7bd3"
format_stamp: "Formatted at 2025-10-26 15:08:03 on dist-test-slave-8pgz"
server_key: "7f4f543e4bd862bfb62525624fa4c5f7"
server_key_iv: "fc0ae84b8b08b78c92538c2a2db7a663"
server_key_version: "encryptionkey@0"
I20251026 15:08:03.916070 19485 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:03.927376 19485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:03.929474 19485 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:03.929590 19485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:03.929989 19485 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:03.930341 19485 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:03.930377 19485 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:03.930437 19485 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:03.930473 19485 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:03.936434 19485 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:45049
I20251026 15:08:03.936480 19614 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:45049 every 8 connection(s)
I20251026 15:08:03.936875 19485 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
Oct 26 15:08:03 dist-test-slave-8pgz krb5kdc[19371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491283, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:03.948062 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:03.937684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:58901 (local address 127.14.212.126:46509)
1026 15:08:03.937757 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:03.937759 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:03.937989 (+ 230us) server_negotiation.cc:408] Connection header received
1026 15:08:03.938773 (+ 784us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:03.938774 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:03.938788 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:03.938856 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:03.940003 (+ 1147us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:03.940769 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:03.942111 (+ 1342us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:03.942312 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:03.944944 (+ 2632us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:03.944949 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:03.944950 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:03.944958 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:03.946711 (+ 1753us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:03.947163 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:03.947164 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:03.947164 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:03.947192 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:03.947382 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:03.947383 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:03.947383 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:03.947492 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:03.947531 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:03.947866 (+ 335us) server_negotiation.cc:300] Negotiation successful
1026 15:08:03.947968 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251026 15:08:03.948495 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:03.937608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46509 (local address 127.14.212.65:58901)
1026 15:08:03.937846 (+ 238us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:03.937883 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:03.938663 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:03.938923 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:03.938929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:03.939444 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:03.939895 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:03.939900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:03.940880 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:03.940881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:03.941942 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:03.941945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:03.942113 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:03.942742 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:03.942766 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:03.944733 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:03.946808 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:03.946815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:03.946817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:03.947080 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:03.947253 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:03.947254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:03.947255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:03.947326 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:03.947569 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:03.947574 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:03.947723 (+ 149us) client_negotiation.cc:769] Sending connection context
1026 15:08:03.948173 (+ 450us) client_negotiation.cc:240] Negotiation successful
1026 15:08:03.948312 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
I20251026 15:08:03.952461 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 19485
I20251026 15:08:03.952567 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:08:03.952749 15185 external_mini_cluster.cc:1468] Setting key 55657e1461f248959c0f0f48658eefdd
I20251026 15:08:03.953377 19615 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46509
I20251026 15:08:03.953481 19615 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:03.953707 19615 heartbeater.cc:507] Master 127.14.212.126:46509 requested a full tablet report, sending...
I20251026 15:08:03.954252 19419 ts_manager.cc:194] Registered new tserver with Master: 6edf9ea24b05406f8d65c4745abb7bd3 (127.14.212.65:45049)
I20251026 15:08:03.954921 19419 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:58901
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:03.999552 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46509
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:04.078012 19622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:04.078192 19622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:04.078217 19622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:04.079659 19622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:04.079723 19622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:04.079751 19622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:08:04.081326 19622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:04.081600 19622 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:04.081795 19622 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:04.084676 19628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:04.084671 19627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:04.084708 19622 server_base.cc:1047] running on GCE node
W20251026 15:08:04.084676 19630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:04.085100 19622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:04.085336 19622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:04.086459 19622 hybrid_clock.cc:648] HybridClock initialized: now 1761491284086447 us; error 31 us; skew 500 ppm
Oct 26 15:08:04 dist-test-slave-8pgz krb5kdc[19371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491284, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:04.088817 19622 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:08:04.089340 19622 webserver.cc:492] Webserver started at http://127.14.212.66:42577/ using document root <none> and password file <none>
I20251026 15:08:04.089512 19622 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:04.089545 19622 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:04.089632 19622 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:04.090696 19622 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "8ceade603e364c988627ecea5651eeaf"
format_stamp: "Formatted at 2025-10-26 15:08:04 on dist-test-slave-8pgz"
server_key: "2c936a0e823ebd8e3a2b9ca609c494b6"
server_key_iv: "25162f500768d4321e4b7c2db842781e"
server_key_version: "encryptionkey@0"
I20251026 15:08:04.090981 19622 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "8ceade603e364c988627ecea5651eeaf"
format_stamp: "Formatted at 2025-10-26 15:08:04 on dist-test-slave-8pgz"
server_key: "2c936a0e823ebd8e3a2b9ca609c494b6"
server_key_iv: "25162f500768d4321e4b7c2db842781e"
server_key_version: "encryptionkey@0"
I20251026 15:08:04.092235 19622 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:08:04.092919 19637 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:04.093109 19622 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:04.093185 19622 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "8ceade603e364c988627ecea5651eeaf"
format_stamp: "Formatted at 2025-10-26 15:08:04 on dist-test-slave-8pgz"
server_key: "2c936a0e823ebd8e3a2b9ca609c494b6"
server_key_iv: "25162f500768d4321e4b7c2db842781e"
server_key_version: "encryptionkey@0"
I20251026 15:08:04.093236 19622 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:04.102474 19622 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:04.104668 19622 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:04.104785 19622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:04.105003 19622 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:04.105290 19622 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:04.105322 19622 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:04.105346 19622 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:04.105358 19622 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:04.111415 19622 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:33645
I20251026 15:08:04.111495 19751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:33645 every 8 connection(s)
I20251026 15:08:04.111768 19622 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:08:04.113744 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 19622
I20251026 15:08:04.113844 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:08:04.113999 15185 external_mini_cluster.cc:1468] Setting key 06b94024a81497a41001b68c23eebe9c
Oct 26 15:08:04 dist-test-slave-8pgz krb5kdc[19371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491284, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:04.122328 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:04.112551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:52955 (local address 127.14.212.126:46509)
1026 15:08:04.112701 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:04.112703 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:04.112832 (+ 129us) server_negotiation.cc:408] Connection header received
1026 15:08:04.113385 (+ 553us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:04.113387 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:04.113400 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:04.113453 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:04.114291 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:04.114896 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:04.116340 (+ 1444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:04.116555 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:04.119016 (+ 2461us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:04.119021 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:04.119022 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:04.119031 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:04.120935 (+ 1904us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:04.121430 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:04.121430 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:04.121431 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:04.121451 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:04.121662 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:04.121662 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:04.121662 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:04.121748 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:04.121794 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:04.122185 (+ 391us) server_negotiation.cc:300] Negotiation successful
1026 15:08:04.122260 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
I20251026 15:08:04.122563 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:04.112472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46509 (local address 127.14.212.66:52955)
1026 15:08:04.112715 (+ 243us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:04.112750 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:04.113287 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:04.113526 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:04.113530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:04.113863 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:04.114191 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:04.114195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:04.114987 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:04.114988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:04.116227 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:04.116231 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:04.116386 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:04.116954 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:04.116973 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:04.118791 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:04.121031 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:04.121039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:04.121043 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:04.121352 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:04.121509 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:04.121510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:04.121511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:04.121606 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:04.121804 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:04.121808 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:04.121988 (+ 180us) client_negotiation.cc:769] Sending connection context
1026 15:08:04.122160 (+ 172us) client_negotiation.cc:240] Negotiation successful
1026 15:08:04.122319 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
I20251026 15:08:04.123122 19752 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46509
I20251026 15:08:04.123217 19752 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:04.123392 19752 heartbeater.cc:507] Master 127.14.212.126:46509 requested a full tablet report, sending...
I20251026 15:08:04.123757 19419 ts_manager.cc:194] Registered new tserver with Master: 8ceade603e364c988627ecea5651eeaf (127.14.212.66:33645)
I20251026 15:08:04.124136 19419 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:52955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:04.160339 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46509
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:04.240485 19759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:04.240664 19759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:04.240686 19759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:04.242089 19759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:04.242137 19759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:04.242162 19759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:08:04.243731 19759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:04.243943 19759 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:04.244096 19759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:04.246632 19767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:04.246628 19765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:04.246635 19764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:04.246956 19759 server_base.cc:1047] running on GCE node
I20251026 15:08:04.247105 19759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:04.247318 19759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:04.248451 19759 hybrid_clock.cc:648] HybridClock initialized: now 1761491284248438 us; error 30 us; skew 500 ppm
Oct 26 15:08:04 dist-test-slave-8pgz krb5kdc[19371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491284, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:04.250840 19759 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:08:04.251358 19759 webserver.cc:492] Webserver started at http://127.14.212.67:41321/ using document root <none> and password file <none>
I20251026 15:08:04.251538 19759 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:04.251572 19759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:04.251679 19759 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:04.252620 19759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "0c5dc86c4d7c4dc9b037b4987d83d8c5"
format_stamp: "Formatted at 2025-10-26 15:08:04 on dist-test-slave-8pgz"
server_key: "179906491faa5da4f30c1ad17a452ccd"
server_key_iv: "1c347aff581ffbb5fc64f59b9298ede9"
server_key_version: "encryptionkey@0"
I20251026 15:08:04.252918 19759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "0c5dc86c4d7c4dc9b037b4987d83d8c5"
format_stamp: "Formatted at 2025-10-26 15:08:04 on dist-test-slave-8pgz"
server_key: "179906491faa5da4f30c1ad17a452ccd"
server_key_iv: "1c347aff581ffbb5fc64f59b9298ede9"
server_key_version: "encryptionkey@0"
I20251026 15:08:04.254052 19759 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251026 15:08:04.254806 19774 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:04.254987 19759 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:04.255061 19759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "0c5dc86c4d7c4dc9b037b4987d83d8c5"
format_stamp: "Formatted at 2025-10-26 15:08:04 on dist-test-slave-8pgz"
server_key: "179906491faa5da4f30c1ad17a452ccd"
server_key_iv: "1c347aff581ffbb5fc64f59b9298ede9"
server_key_version: "encryptionkey@0"
I20251026 15:08:04.255139 19759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:04.270656 19759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:04.272511 19759 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:04.272650 19759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:04.272867 19759 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:04.273198 19759 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:04.273232 19759 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:04.273255 19759 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:04.273283 19759 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:04.278931 19759 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:41393
I20251026 15:08:04.279001 19888 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:41393 every 8 connection(s)
I20251026 15:08:04.279348 19759 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
Oct 26 15:08:04 dist-test-slave-8pgz krb5kdc[19371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491284, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:04.285044 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 19759
I20251026 15:08:04.285156 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:08:04.285318 15185 external_mini_cluster.cc:1468] Setting key 3db32c633580778ed92630fb506f06e7
I20251026 15:08:04.288228 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:04.280085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:45117 (local address 127.14.212.126:46509)
1026 15:08:04.280169 (+ 84us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:04.280171 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:04.280304 (+ 133us) server_negotiation.cc:408] Connection header received
1026 15:08:04.280759 (+ 455us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:04.280760 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:04.280769 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:04.280829 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:04.281596 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:04.282156 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:04.283343 (+ 1187us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:04.283474 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:04.285617 (+ 2143us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:04.285622 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:04.285622 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:04.285630 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:04.287084 (+ 1454us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:04.287447 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:04.287448 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:04.287448 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:04.287473 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:04.287644 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:04.287645 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:04.287646 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:04.287738 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:04.287798 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:04.288051 (+ 253us) server_negotiation.cc:300] Negotiation successful
1026 15:08:04.288138 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251026 15:08:04.288378 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:04.280037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46509 (local address 127.14.212.67:45117)
1026 15:08:04.280191 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:04.280223 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:04.280687 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:04.280881 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:04.280883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:04.281196 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:04.281480 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:04.281483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:04.282265 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:04.282267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:04.283225 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:04.283228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:04.283350 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:04.283899 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:04.283923 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:04.285540 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:04.287161 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:04.287167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:04.287170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:04.287377 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:04.287528 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:04.287528 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:04.287529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:04.287590 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:04.287792 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:04.287796 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:04.287929 (+ 133us) client_negotiation.cc:769] Sending connection context
1026 15:08:04.288028 (+ 99us) client_negotiation.cc:240] Negotiation successful
1026 15:08:04.288172 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20251026 15:08:04.288798 19889 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46509
I20251026 15:08:04.288899 19889 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:04.289065 19889 heartbeater.cc:507] Master 127.14.212.126:46509 requested a full tablet report, sending...
I20251026 15:08:04.289412 19419 ts_manager.cc:194] Registered new tserver with Master: 0c5dc86c4d7c4dc9b037b4987d83d8c5 (127.14.212.67:41393)
I20251026 15:08:04.289875 19419 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:45117
I20251026 15:08:04.297300 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 26 15:08:04 dist-test-slave-8pgz krb5kdc[19371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 26 15:08:04 dist-test-slave-8pgz krb5kdc[19371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.65@KRBTEST.COM
I20251026 15:08:04.319054 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:04.311752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.14.212.65:45049)
1026 15:08:04.311939 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:04.311946 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:04.311955 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:08:04.311980 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:04.311982 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:04.311996 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:04.312063 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:04.312828 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:04.313549 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:04.314117 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:04.314356 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:04.316434 (+ 2078us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:04.316447 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:04.316447 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:04.316454 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:04.318038 (+ 1584us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:04.318345 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:04.318348 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:04.318348 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:04.318379 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:04.318545 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:04.318546 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:04.318546 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:04.318646 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:04.318788 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:04.318851 (+ 63us) server_negotiation.cc:300] Negotiation successful
1026 15:08:04.318940 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
Oct 26 15:08:04 dist-test-slave-8pgz krb5kdc[19371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:04.326666 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:04.320454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.14.212.126:46509)
1026 15:08:04.320516 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:04.320518 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:04.320529 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:08:04.320580 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:04.320581 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:04.320592 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:04.320647 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:04.321400 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:04.321880 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:04.322413 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:04.322603 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:04.324504 (+ 1901us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:04.324510 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:04.324511 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:04.324518 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:04.325693 (+ 1175us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:04.326053 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:04.326054 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:04.326054 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:04.326080 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:04.326222 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:04.326222 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:04.326222 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:04.326329 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:04.326442 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:04.326525 (+ 83us) server_negotiation.cc:300] Negotiation successful
1026 15:08:04.326599 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20251026 15:08:04.326889 19419 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46352
I20251026 15:08:04.327244 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 19485
I20251026 15:08:04.332773 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 19622
I20251026 15:08:04.337308 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 19759
I20251026 15:08:04.341691 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 19388
2025-10-26T15:08:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2878 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:08:04 dist-test-slave-8pgz krb5kdc[19902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:08:04 dist-test-slave-8pgz krb5kdc[19902](info): set up 2 sockets
Oct 26 15:08:04 dist-test-slave-8pgz krb5kdc[19902](info): commencing operation
krb5kdc: starting...
W20251026 15:08:06.331920 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.971s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:08:06 dist-test-slave-8pgz krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:08:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:06.467332 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:40679
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:40679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:06.547612 19918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:06.547808 19918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:06.547853 19918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:06.549276 19918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:06.549324 19918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:06.549345 19918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:06.549364 19918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:06.549381 19918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:08:06.550989 19918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:40679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:40679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:06.551250 19918 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:06.551445 19918 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:06.553910 19923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:06.553988 19918 server_base.cc:1047] running on GCE node
W20251026 15:08:06.553925 19924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:06.553910 19926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:06.554431 19918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:06.554694 19918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:06.555845 19918 hybrid_clock.cc:648] HybridClock initialized: now 1761491286555823 us; error 38 us; skew 500 ppm
Oct 26 15:08:06 dist-test-slave-8pgz krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491286, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:06.558215 19918 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:08:06.558562 19918 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:06.558609 19918 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:06.558708 19918 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:06.559702 19918 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "cfb15d291c2d4121992bf44b3378c0c5"
format_stamp: "Formatted at 2025-10-26 15:08:06 on dist-test-slave-8pgz"
server_key: "a4abc213858f0f4f9edabe40c4db4e2c"
server_key_iv: "8ff4d03778b5a50280d87665c9651e4a"
server_key_version: "encryptionkey@0"
I20251026 15:08:06.559979 19918 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "cfb15d291c2d4121992bf44b3378c0c5"
format_stamp: "Formatted at 2025-10-26 15:08:06 on dist-test-slave-8pgz"
server_key: "a4abc213858f0f4f9edabe40c4db4e2c"
server_key_iv: "8ff4d03778b5a50280d87665c9651e4a"
server_key_version: "encryptionkey@0"
I20251026 15:08:06.561087 19918 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:08:06.561791 19932 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:06.561978 19918 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:06.562049 19918 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "cfb15d291c2d4121992bf44b3378c0c5"
format_stamp: "Formatted at 2025-10-26 15:08:06 on dist-test-slave-8pgz"
server_key: "a4abc213858f0f4f9edabe40c4db4e2c"
server_key_iv: "8ff4d03778b5a50280d87665c9651e4a"
server_key_version: "encryptionkey@0"
I20251026 15:08:06.562103 19918 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:06.571509 19918 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:06.578814 19918 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:06.578953 19918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:06.583276 19918 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:40679
I20251026 15:08:06.583349 19984 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:40679 every 8 connection(s)
I20251026 15:08:06.583596 19918 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:06.584201 19985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:06.586699 19985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5: Bootstrap starting.
I20251026 15:08:06.587291 19985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:06.587530 19985 log.cc:826] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:06.588261 19985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5: No bootstrap required, opened a new log
I20251026 15:08:06.589473 19985 raft_consensus.cc:359] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb15d291c2d4121992bf44b3378c0c5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 40679 } }
I20251026 15:08:06.589599 19985 raft_consensus.cc:385] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:06.589632 19985 raft_consensus.cc:740] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb15d291c2d4121992bf44b3378c0c5, State: Initialized, Role: FOLLOWER
I20251026 15:08:06.589762 19985 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb15d291c2d4121992bf44b3378c0c5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 40679 } }
I20251026 15:08:06.589821 19985 raft_consensus.cc:399] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:06.589856 19985 raft_consensus.cc:493] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:06.589902 19985 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:06.590476 19985 raft_consensus.cc:515] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb15d291c2d4121992bf44b3378c0c5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 40679 } }
I20251026 15:08:06.590574 19985 leader_election.cc:304] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfb15d291c2d4121992bf44b3378c0c5; no voters:
I20251026 15:08:06.590725 19985 leader_election.cc:290] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:08:06.590795 19988 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:06.590936 19985 sys_catalog.cc:565] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:06.590965 19988 raft_consensus.cc:697] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [term 1 LEADER]: Becoming Leader. State: Replica: cfb15d291c2d4121992bf44b3378c0c5, State: Running, Role: LEADER
I20251026 15:08:06.591051 19988 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb15d291c2d4121992bf44b3378c0c5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 40679 } }
I20251026 15:08:06.591410 19989 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfb15d291c2d4121992bf44b3378c0c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb15d291c2d4121992bf44b3378c0c5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 40679 } } }
I20251026 15:08:06.591442 19990 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfb15d291c2d4121992bf44b3378c0c5. Latest consensus state: current_term: 1 leader_uuid: "cfb15d291c2d4121992bf44b3378c0c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfb15d291c2d4121992bf44b3378c0c5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 40679 } } }
I20251026 15:08:06.591524 19989 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:06.591589 19990 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:06.591770 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 19918
I20251026 15:08:06.591821 19996 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:06.591908 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:08:06.592075 15185 external_mini_cluster.cc:1468] Setting key 8e81e839afa52565b4f0946aeef16406
I20251026 15:08:06.592519 19996 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:08:06.594013 19996 catalog_manager.cc:1357] Generated new cluster ID: 50fe693e3640406391dd7bf7d0fcb450
I20251026 15:08:06.594069 19996 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 26 15:08:06 dist-test-slave-8pgz krb5kdc[19902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:06.599035 19996 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:08:06.599768 19996 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:06.602010 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:06.592651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.14.212.126:40679)
1026 15:08:06.592814 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:06.592826 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:06.593329 (+ 503us) server_negotiation.cc:408] Connection header received
1026 15:08:06.593666 (+ 337us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:06.593672 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:06.593722 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:06.593854 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:06.594680 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:06.595448 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:06.595990 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:06.596250 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:06.598383 (+ 2133us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:06.598408 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:06.598409 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:06.598434 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:06.600649 (+ 2215us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:06.601013 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:06.601016 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:06.601016 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:06.601080 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:06.601227 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:06.601228 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:06.601229 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:06.601456 (+ 227us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:06.601611 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:06.601687 (+ 76us) server_negotiation.cc:300] Negotiation successful
1026 15:08:06.601829 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20251026 15:08:06.604321 19996 catalog_manager.cc:6022] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5: Generated new TSK 0
I20251026 15:08:06.604496 19996 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:06.654589 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:40679
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:06.737279 20014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:06.737483 20014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:06.737525 20014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:06.739071 20014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:06.739127 20014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:06.739173 20014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:08:06.740734 20014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:40679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:06.741019 20014 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:06.741230 20014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:06.743885 20019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:06.743904 20022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:06.743916 20020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:06.744191 20014 server_base.cc:1047] running on GCE node
I20251026 15:08:06.744405 20014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:06.744632 20014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:06.745793 20014 hybrid_clock.cc:648] HybridClock initialized: now 1761491286745777 us; error 37 us; skew 500 ppm
Oct 26 15:08:06 dist-test-slave-8pgz krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491286, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:06.748358 20014 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:08:06.748739 20014 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:06.748795 20014 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:06.748968 20014 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:06.750236 20014 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "97ecb53af2ef49379a8bbf3fb84aa471"
format_stamp: "Formatted at 2025-10-26 15:08:06 on dist-test-slave-8pgz"
server_key: "c47887d73217c35484d5b572e8bc8518"
server_key_iv: "4bef14a2bf5c43d553b00c9b92065ad9"
server_key_version: "encryptionkey@0"
I20251026 15:08:06.750607 20014 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "97ecb53af2ef49379a8bbf3fb84aa471"
format_stamp: "Formatted at 2025-10-26 15:08:06 on dist-test-slave-8pgz"
server_key: "c47887d73217c35484d5b572e8bc8518"
server_key_iv: "4bef14a2bf5c43d553b00c9b92065ad9"
server_key_version: "encryptionkey@0"
I20251026 15:08:06.752091 20014 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:06.752938 20028 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:06.753144 20014 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:06.753233 20014 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "97ecb53af2ef49379a8bbf3fb84aa471"
format_stamp: "Formatted at 2025-10-26 15:08:06 on dist-test-slave-8pgz"
server_key: "c47887d73217c35484d5b572e8bc8518"
server_key_iv: "4bef14a2bf5c43d553b00c9b92065ad9"
server_key_version: "encryptionkey@0"
I20251026 15:08:06.753306 20014 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:06.763492 20014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:06.765400 20014 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:06.765506 20014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:06.765672 20014 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:06.765965 20014 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:06.765997 20014 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:06.766021 20014 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:06.766038 20014 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:06.771705 20014 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:43985
I20251026 15:08:06.771770 20142 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:43985 every 8 connection(s)
I20251026 15:08:06.771988 20014 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
Oct 26 15:08:06 dist-test-slave-8pgz krb5kdc[19902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491286, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:06.779469 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 20014
I20251026 15:08:06.779578 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:08:06.779748 15185 external_mini_cluster.cc:1468] Setting key ee52adfd183de97eaeff9f58c296af32
I20251026 15:08:06.780452 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:06.772624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:48703 (local address 127.14.212.126:40679)
1026 15:08:06.772688 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:06.772691 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:06.772836 (+ 145us) server_negotiation.cc:408] Connection header received
1026 15:08:06.773354 (+ 518us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:06.773356 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:06.773369 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:06.773428 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:06.774509 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:06.775007 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:06.775570 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:06.775745 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:06.777733 (+ 1988us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:06.777737 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:06.777737 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:06.777744 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:06.779101 (+ 1357us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:06.779551 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:06.779553 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:06.779554 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:06.779580 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:06.779780 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:06.779781 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:06.779782 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:06.779901 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:06.779971 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:06.780253 (+ 282us) server_negotiation.cc:300] Negotiation successful
1026 15:08:06.780345 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251026 15:08:06.780714 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:06.772564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:40679 (local address 127.14.212.65:48703)
1026 15:08:06.772729 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:06.772761 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:06.773249 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:06.773495 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:06.773498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:06.773794 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:06.774378 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:06.774381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:06.775091 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:06.775092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:06.775490 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:06.775491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:06.775606 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:06.776191 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:06.776210 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:06.777636 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:06.779177 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:06.779183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:06.779186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:06.779468 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:06.779648 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:06.779649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:06.779650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:06.779720 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:06.779959 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:06.779964 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:06.780111 (+ 147us) client_negotiation.cc:769] Sending connection context
1026 15:08:06.780230 (+ 119us) client_negotiation.cc:240] Negotiation successful
1026 15:08:06.780388 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20251026 15:08:06.781174 20143 heartbeater.cc:344] Connected to a master server at 127.14.212.126:40679
I20251026 15:08:06.781260 20143 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:06.781440 20143 heartbeater.cc:507] Master 127.14.212.126:40679 requested a full tablet report, sending...
I20251026 15:08:06.782042 19949 ts_manager.cc:194] Registered new tserver with Master: 97ecb53af2ef49379a8bbf3fb84aa471 (127.14.212.65:43985)
I20251026 15:08:06.782722 19949 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:48703
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:06.825034 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:40679
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:06.906934 20150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:06.907198 20150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:06.907238 20150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:06.909188 20150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:06.909257 20150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:06.909291 20150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:08:06.911054 20150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:40679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:06.911309 20150 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:06.911520 20150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:06.914445 20155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:06.914588 20150 server_base.cc:1047] running on GCE node
W20251026 15:08:06.914438 20156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:06.914438 20158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:06.914909 20150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:06.915128 20150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:06.916286 20150 hybrid_clock.cc:648] HybridClock initialized: now 1761491286916279 us; error 30 us; skew 500 ppm
Oct 26 15:08:06 dist-test-slave-8pgz krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491286, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:06.918676 20150 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:08:06.919013 20150 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:06.919064 20150 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:06.919188 20150 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:06.920189 20150 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "70d6f05162774770ac0d62fc64d62e57"
format_stamp: "Formatted at 2025-10-26 15:08:06 on dist-test-slave-8pgz"
server_key: "f6a250cb3240a8cbde0e708bc67322f1"
server_key_iv: "85a48a84ee61a014c5beb65591167266"
server_key_version: "encryptionkey@0"
I20251026 15:08:06.920490 20150 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "70d6f05162774770ac0d62fc64d62e57"
format_stamp: "Formatted at 2025-10-26 15:08:06 on dist-test-slave-8pgz"
server_key: "f6a250cb3240a8cbde0e708bc67322f1"
server_key_iv: "85a48a84ee61a014c5beb65591167266"
server_key_version: "encryptionkey@0"
I20251026 15:08:06.921679 20150 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:08:06.922379 20164 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:06.922581 20150 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251026 15:08:06.922664 20150 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "70d6f05162774770ac0d62fc64d62e57"
format_stamp: "Formatted at 2025-10-26 15:08:06 on dist-test-slave-8pgz"
server_key: "f6a250cb3240a8cbde0e708bc67322f1"
server_key_iv: "85a48a84ee61a014c5beb65591167266"
server_key_version: "encryptionkey@0"
I20251026 15:08:06.922727 20150 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:06.939641 20150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:06.941728 20150 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:06.941882 20150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:06.942124 20150 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:06.942484 20150 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:06.942523 20150 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:06.942559 20150 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:06.942588 20150 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:06.948529 20150 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:44003
I20251026 15:08:06.948583 20278 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:44003 every 8 connection(s)
I20251026 15:08:06.948812 20150 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:08:06.949833 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 20150
I20251026 15:08:06.949942 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:08:06.950109 15185 external_mini_cluster.cc:1468] Setting key dc887ae1186a82e1f4245aa1ec5908db
Oct 26 15:08:06 dist-test-slave-8pgz krb5kdc[19902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491286, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:06.957271 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:06.949513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:59679 (local address 127.14.212.126:40679)
1026 15:08:06.949593 (+ 80us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:06.949595 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:06.949710 (+ 115us) server_negotiation.cc:408] Connection header received
1026 15:08:06.950263 (+ 553us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:06.950264 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:06.950279 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:06.950328 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:06.951405 (+ 1077us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:06.951911 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:06.952470 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:06.952642 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:06.954564 (+ 1922us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:06.954571 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:06.954571 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:06.954578 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:06.956105 (+ 1527us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:06.956464 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:06.956465 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:06.956465 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:06.956490 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:06.956658 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:06.956659 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:06.956659 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:06.956757 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:06.956819 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:06.957082 (+ 263us) server_negotiation.cc:300] Negotiation successful
1026 15:08:06.957179 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
I20251026 15:08:06.957412 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:06.949427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:40679 (local address 127.14.212.66:59679)
1026 15:08:06.949595 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:06.949630 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:06.950162 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:06.950385 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:06.950409 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:06.950748 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:06.951312 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:06.951316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:06.951990 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:06.951991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:06.952396 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:06.952398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:06.952514 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:06.953099 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:06.953116 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:06.954485 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:06.956190 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:06.956196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:06.956199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:06.956397 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:06.956544 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:06.956545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:06.956545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:06.956604 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:06.956812 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:06.956816 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:06.956944 (+ 128us) client_negotiation.cc:769] Sending connection context
1026 15:08:06.957064 (+ 120us) client_negotiation.cc:240] Negotiation successful
1026 15:08:06.957213 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20251026 15:08:06.957947 20279 heartbeater.cc:344] Connected to a master server at 127.14.212.126:40679
I20251026 15:08:06.958041 20279 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:06.958204 20279 heartbeater.cc:507] Master 127.14.212.126:40679 requested a full tablet report, sending...
I20251026 15:08:06.958611 19949 ts_manager.cc:194] Registered new tserver with Master: 70d6f05162774770ac0d62fc64d62e57 (127.14.212.66:44003)
I20251026 15:08:06.959048 19949 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:59679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:06.997608 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:40679
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:07.079357 20286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:07.079538 20286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:07.079562 20286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:07.080962 20286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:07.081002 20286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:07.081028 20286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:08:07.082588 20286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:40679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:07.082775 20286 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:07.082931 20286 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:07.085438 20291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:07.085454 20294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:07.085569 20286 server_base.cc:1047] running on GCE node
W20251026 15:08:07.085713 20292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:07.085870 20286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:07.086076 20286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:07.087213 20286 hybrid_clock.cc:648] HybridClock initialized: now 1761491287087198 us; error 27 us; skew 500 ppm
Oct 26 15:08:07 dist-test-slave-8pgz krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491287, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:07.089594 20286 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:08:07.089918 20286 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:07.089973 20286 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:07.090101 20286 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:07.091116 20286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "2053323e7f8c412596907543f9ad62d3"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "0049f26f61f0f7dfc6cfa83106f7cca5"
server_key_iv: "47c1e6b5fd72c13b1a71e6e07ceda5c5"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.091423 20286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "2053323e7f8c412596907543f9ad62d3"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "0049f26f61f0f7dfc6cfa83106f7cca5"
server_key_iv: "47c1e6b5fd72c13b1a71e6e07ceda5c5"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.092626 20286 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:08:07.093429 20300 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:07.093608 20286 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251026 15:08:07.093688 20286 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "2053323e7f8c412596907543f9ad62d3"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "0049f26f61f0f7dfc6cfa83106f7cca5"
server_key_iv: "47c1e6b5fd72c13b1a71e6e07ceda5c5"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.093737 20286 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:07.108484 20286 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:07.111117 20286 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:07.111258 20286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:07.111465 20286 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:07.111753 20286 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:07.111783 20286 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:07.111807 20286 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:07.111820 20286 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:07.117734 20286 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:36643
I20251026 15:08:07.117776 20413 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:36643 every 8 connection(s)
I20251026 15:08:07.118049 20286 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:08:07.122681 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 20286
I20251026 15:08:07.122758 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:08:07.122905 15185 external_mini_cluster.cc:1468] Setting key 2a63d8454bdaddf5ece5821b2cdde68f
Oct 26 15:08:07 dist-test-slave-8pgz krb5kdc[19902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491287, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:07.127316 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.118754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:40943 (local address 127.14.212.126:40679)
1026 15:08:07.118807 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.118809 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.119028 (+ 219us) server_negotiation.cc:408] Connection header received
1026 15:08:07.119479 (+ 451us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.119481 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.119491 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.119556 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:07.120705 (+ 1149us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.121218 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.121892 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.122021 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.124448 (+ 2427us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:07.124452 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:07.124453 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:07.124460 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:07.125900 (+ 1440us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:07.126341 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:07.126344 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:07.126344 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:07.126403 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:07.126624 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:07.126624 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:07.126625 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:07.126727 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:07.126788 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.127124 (+ 336us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.127223 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251026 15:08:07.127431 20416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.118706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:40679 (local address 127.14.212.67:40943)
1026 15:08:07.118914 (+ 208us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:07.118948 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:07.119401 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:07.119675 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:07.119679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:07.120028 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:07.120588 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:07.120591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.121337 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.121338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:07.121761 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:07.121762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.121916 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.122585 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:07.122605 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:07.124360 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:07.125999 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:07.126006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:07.126010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:07.126264 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:07.126472 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:07.126473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:07.126473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:07.126562 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:07.126783 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:07.126789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:07.126960 (+ 171us) client_negotiation.cc:769] Sending connection context
1026 15:08:07.127062 (+ 102us) client_negotiation.cc:240] Negotiation successful
1026 15:08:07.127221 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20251026 15:08:07.127899 20414 heartbeater.cc:344] Connected to a master server at 127.14.212.126:40679
I20251026 15:08:07.128006 20414 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:07.128201 20414 heartbeater.cc:507] Master 127.14.212.126:40679 requested a full tablet report, sending...
I20251026 15:08:07.128628 19949 ts_manager.cc:194] Registered new tserver with Master: 2053323e7f8c412596907543f9ad62d3 (127.14.212.67:36643)
I20251026 15:08:07.128980 19949 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:40943
I20251026 15:08:07.134984 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:08:07.142168 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.136364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.14.212.126:40679)
1026 15:08:07.136413 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.136415 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.136504 (+ 89us) server_negotiation.cc:408] Connection header received
1026 15:08:07.136580 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.136581 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.136590 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.136635 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:07.137355 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.137855 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.138420 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.138564 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.139531 (+ 967us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:07.139536 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:07.139536 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:07.139542 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:07.140917 (+ 1375us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:07.141264 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:07.141265 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:07.141266 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:07.141291 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:07.141510 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:07.141511 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:07.141512 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:07.141637 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:07.141734 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.141954 (+ 220us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.142034 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251026 15:08:07.143913 19949 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:08:07.145226 19949 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:08:07.151324 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.147615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:44003 (local address 127.0.0.1:48460)
1026 15:08:07.147917 (+ 302us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:07.147927 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:07.147978 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:07.148546 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:07.148547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:07.148549 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:07.148753 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:07.148755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.149983 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.149984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:07.150971 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:07.150973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.151053 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.151097 (+ 44us) client_negotiation.cc:769] Sending connection context
1026 15:08:07.151153 (+ 56us) client_negotiation.cc:240] Negotiation successful
1026 15:08:07.151203 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":95,"threads_started":1}
I20251026 15:08:07.151324 20426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.147584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:36643 (local address 127.0.0.1:45944)
1026 15:08:07.147858 (+ 274us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:07.147889 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:07.147978 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:07.148281 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:07.148283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:07.148286 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:07.148569 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:07.148571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.149933 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.149934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:07.150974 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:07.150975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.151055 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.151112 (+ 57us) client_negotiation.cc:769] Sending connection context
1026 15:08:07.151159 (+ 47us) client_negotiation.cc:240] Negotiation successful
1026 15:08:07.151208 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":64,"threads_started":1}
I20251026 15:08:07.151911 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.147725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48460 (local address 127.14.212.66:44003)
1026 15:08:07.148405 (+ 680us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.148413 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.148424 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:08:07.148459 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.148461 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.148476 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.148542 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:07.148844 (+ 302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.149924 (+ 1080us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.151117 (+ 1193us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.151654 (+ 537us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.151719 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.151750 (+ 31us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.151798 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":93,"threads_started":1}
I20251026 15:08:07.152240 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.147687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45944 (local address 127.14.212.67:36643)
1026 15:08:07.148118 (+ 431us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.148124 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.148133 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:08:07.148168 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.148170 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.148183 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.148245 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:07.148666 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.149837 (+ 1171us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.151126 (+ 1289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.152046 (+ 920us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.152113 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.152144 (+ 31us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.152188 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":84,"threads_started":1}
I20251026 15:08:07.152307 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.147782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:43985 (local address 127.0.0.1:37258)
1026 15:08:07.148642 (+ 860us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:07.148653 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:07.148714 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:07.148897 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:07.148898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:07.148900 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:07.149072 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:07.149074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.151351 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.151351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:07.152087 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:07.152090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.152176 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.152180 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:07.152211 (+ 31us) client_negotiation.cc:240] Negotiation successful
1026 15:08:07.152247 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":62,"threads_started":1}
I20251026 15:08:07.152520 20212 tablet_service.cc:1505] Processing CreateTablet for tablet 21580042a60f484490005a8f24260a89 (DEFAULT_TABLE table=test-table [id=b97fb750a8db4850abe85c435cec9ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:07.152845 20212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21580042a60f484490005a8f24260a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:07.152869 20348 tablet_service.cc:1505] Processing CreateTablet for tablet 21580042a60f484490005a8f24260a89 (DEFAULT_TABLE table=test-table [id=b97fb750a8db4850abe85c435cec9ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:07.153141 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.147893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.14.212.65:43985)
1026 15:08:07.148107 (+ 214us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.148124 (+ 17us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.148719 (+ 595us) server_negotiation.cc:408] Connection header received
1026 15:08:07.148787 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.148790 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.148809 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.148891 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:07.149147 (+ 256us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.149843 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.152410 (+ 2567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.152933 (+ 523us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.152980 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.153011 (+ 31us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.153058 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20251026 15:08:07.153223 20348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21580042a60f484490005a8f24260a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:07.153718 20076 tablet_service.cc:1505] Processing CreateTablet for tablet 21580042a60f484490005a8f24260a89 (DEFAULT_TABLE table=test-table [id=b97fb750a8db4850abe85c435cec9ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:07.153975 20076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21580042a60f484490005a8f24260a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:07.155838 20431 tablet_bootstrap.cc:492] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3: Bootstrap starting.
I20251026 15:08:07.156002 20432 tablet_bootstrap.cc:492] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57: Bootstrap starting.
I20251026 15:08:07.156579 20432 tablet_bootstrap.cc:654] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:07.156617 20431 tablet_bootstrap.cc:654] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:07.156770 20433 tablet_bootstrap.cc:492] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471: Bootstrap starting.
I20251026 15:08:07.156965 20432 log.cc:826] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:07.156965 20431 log.cc:826] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:07.157371 20433 tablet_bootstrap.cc:654] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:07.157663 20433 log.cc:826] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:07.157671 20432 tablet_bootstrap.cc:492] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57: No bootstrap required, opened a new log
I20251026 15:08:07.157671 20431 tablet_bootstrap.cc:492] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3: No bootstrap required, opened a new log
I20251026 15:08:07.157747 20431 ts_tablet_manager.cc:1403] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:08:07.157747 20432 ts_tablet_manager.cc:1403] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:08:07.158274 20433 tablet_bootstrap.cc:492] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471: No bootstrap required, opened a new log
I20251026 15:08:07.158316 20433 ts_tablet_manager.cc:1403] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:08:07.159649 20431 raft_consensus.cc:359] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053323e7f8c412596907543f9ad62d3" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36643 } } peers { permanent_uuid: "97ecb53af2ef49379a8bbf3fb84aa471" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43985 } } peers { permanent_uuid: "70d6f05162774770ac0d62fc64d62e57" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44003 } }
I20251026 15:08:07.159797 20431 raft_consensus.cc:385] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:07.159827 20431 raft_consensus.cc:740] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2053323e7f8c412596907543f9ad62d3, State: Initialized, Role: FOLLOWER
I20251026 15:08:07.159857 20432 raft_consensus.cc:359] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053323e7f8c412596907543f9ad62d3" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36643 } } peers { permanent_uuid: "97ecb53af2ef49379a8bbf3fb84aa471" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43985 } } peers { permanent_uuid: "70d6f05162774770ac0d62fc64d62e57" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44003 } }
I20251026 15:08:07.159960 20432 raft_consensus.cc:385] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:07.159890 20433 raft_consensus.cc:359] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053323e7f8c412596907543f9ad62d3" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36643 } } peers { permanent_uuid: "97ecb53af2ef49379a8bbf3fb84aa471" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43985 } } peers { permanent_uuid: "70d6f05162774770ac0d62fc64d62e57" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44003 } }
I20251026 15:08:07.159987 20432 raft_consensus.cc:740] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d6f05162774770ac0d62fc64d62e57, State: Initialized, Role: FOLLOWER
I20251026 15:08:07.159937 20431 consensus_queue.cc:260] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053323e7f8c412596907543f9ad62d3" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36643 } } peers { permanent_uuid: "97ecb53af2ef49379a8bbf3fb84aa471" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43985 } } peers { permanent_uuid: "70d6f05162774770ac0d62fc64d62e57" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44003 } }
I20251026 15:08:07.160009 20433 raft_consensus.cc:385] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:07.160030 20433 raft_consensus.cc:740] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ecb53af2ef49379a8bbf3fb84aa471, State: Initialized, Role: FOLLOWER
I20251026 15:08:07.160070 20432 consensus_queue.cc:260] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053323e7f8c412596907543f9ad62d3" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36643 } } peers { permanent_uuid: "97ecb53af2ef49379a8bbf3fb84aa471" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43985 } } peers { permanent_uuid: "70d6f05162774770ac0d62fc64d62e57" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44003 } }
I20251026 15:08:07.160127 20433 consensus_queue.cc:260] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053323e7f8c412596907543f9ad62d3" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36643 } } peers { permanent_uuid: "97ecb53af2ef49379a8bbf3fb84aa471" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43985 } } peers { permanent_uuid: "70d6f05162774770ac0d62fc64d62e57" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44003 } }
I20251026 15:08:07.160197 20431 ts_tablet_manager.cc:1434] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:08:07.160307 20432 ts_tablet_manager.cc:1434] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251026 15:08:07.160338 20279 heartbeater.cc:499] Master 127.14.212.126:40679 was elected leader, sending a full tablet report...
I20251026 15:08:07.160430 20414 heartbeater.cc:499] Master 127.14.212.126:40679 was elected leader, sending a full tablet report...
I20251026 15:08:07.160655 20433 ts_tablet_manager.cc:1434] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:08:07.161067 20143 heartbeater.cc:499] Master 127.14.212.126:40679 was elected leader, sending a full tablet report...
W20251026 15:08:07.199357 20280 tablet.cc:2378] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 15:08:07.272711 20144 tablet.cc:2378] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:08:07.361701 20438 raft_consensus.cc:493] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:08:07.361822 20438 raft_consensus.cc:515] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053323e7f8c412596907543f9ad62d3" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36643 } } peers { permanent_uuid: "97ecb53af2ef49379a8bbf3fb84aa471" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43985 } } peers { permanent_uuid: "70d6f05162774770ac0d62fc64d62e57" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44003 } }
I20251026 15:08:07.362162 20438 leader_election.cc:290] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2053323e7f8c412596907543f9ad62d3 (127.14.212.67:36643), 97ecb53af2ef49379a8bbf3fb84aa471 (127.14.212.65:43985)
I20251026 15:08:07.364404 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.362336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:36643 (local address 127.14.212.66:42395)
1026 15:08:07.362435 (+ 99us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:07.362450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:07.362537 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:07.362708 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:07.362710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:07.362712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:07.362961 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:07.362964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.363629 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.363630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:07.364233 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:07.364235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.364297 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.364300 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:08:07.364324 (+ 24us) client_negotiation.cc:240] Negotiation successful
1026 15:08:07.364351 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20251026 15:08:07.364600 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.362468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:43985 (local address 127.14.212.66:56691)
1026 15:08:07.362665 (+ 197us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:07.362674 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:07.362728 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:07.362860 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:07.362861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:07.362862 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:07.363089 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:07.363091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.363759 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.363760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:07.364376 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:07.364377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.364449 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.364452 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:08:07.364485 (+ 33us) client_negotiation.cc:240] Negotiation successful
1026 15:08:07.364519 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":113,"threads_started":1}
I20251026 15:08:07.364895 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.362425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:42395 (local address 127.14.212.67:36643)
1026 15:08:07.362475 (+ 50us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.362478 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.362566 (+ 88us) server_negotiation.cc:408] Connection header received
1026 15:08:07.362609 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.362610 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.362621 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.362683 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:07.363036 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.363549 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.364314 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.364795 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.364803 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.364822 (+ 19us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.364844 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251026 15:08:07.365108 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.362578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:56691 (local address 127.14.212.65:43985)
1026 15:08:07.362623 (+ 45us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.362626 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.362740 (+ 114us) server_negotiation.cc:408] Connection header received
1026 15:08:07.362773 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.362774 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.362785 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.362854 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:07.363188 (+ 334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.363666 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.364474 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.364961 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.364969 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.364996 (+ 27us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.365034 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251026 15:08:07.365214 20368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21580042a60f484490005a8f24260a89" candidate_uuid: "70d6f05162774770ac0d62fc64d62e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2053323e7f8c412596907543f9ad62d3" is_pre_election: true
I20251026 15:08:07.365399 20368 raft_consensus.cc:2468] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70d6f05162774770ac0d62fc64d62e57 in term 0.
I20251026 15:08:07.365377 20096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21580042a60f484490005a8f24260a89" candidate_uuid: "70d6f05162774770ac0d62fc64d62e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ecb53af2ef49379a8bbf3fb84aa471" is_pre_election: true
I20251026 15:08:07.365494 20096 raft_consensus.cc:2468] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70d6f05162774770ac0d62fc64d62e57 in term 0.
I20251026 15:08:07.365646 20167 leader_election.cc:304] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2053323e7f8c412596907543f9ad62d3, 70d6f05162774770ac0d62fc64d62e57; no voters:
I20251026 15:08:07.365768 20438 raft_consensus.cc:2804] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:08:07.365825 20438 raft_consensus.cc:493] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:08:07.365859 20438 raft_consensus.cc:3060] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:07.366541 20438 raft_consensus.cc:515] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053323e7f8c412596907543f9ad62d3" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36643 } } peers { permanent_uuid: "97ecb53af2ef49379a8bbf3fb84aa471" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43985 } } peers { permanent_uuid: "70d6f05162774770ac0d62fc64d62e57" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44003 } }
I20251026 15:08:07.366667 20438 leader_election.cc:290] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [CANDIDATE]: Term 1 election: Requested vote from peers 2053323e7f8c412596907543f9ad62d3 (127.14.212.67:36643), 97ecb53af2ef49379a8bbf3fb84aa471 (127.14.212.65:43985)
I20251026 15:08:07.366830 20368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21580042a60f484490005a8f24260a89" candidate_uuid: "70d6f05162774770ac0d62fc64d62e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2053323e7f8c412596907543f9ad62d3"
I20251026 15:08:07.366904 20368 raft_consensus.cc:3060] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:07.366848 20096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21580042a60f484490005a8f24260a89" candidate_uuid: "70d6f05162774770ac0d62fc64d62e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ecb53af2ef49379a8bbf3fb84aa471"
I20251026 15:08:07.366955 20096 raft_consensus.cc:3060] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:07.367547 20368 raft_consensus.cc:2468] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70d6f05162774770ac0d62fc64d62e57 in term 1.
I20251026 15:08:07.367678 20096 raft_consensus.cc:2468] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70d6f05162774770ac0d62fc64d62e57 in term 1.
I20251026 15:08:07.367704 20167 leader_election.cc:304] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2053323e7f8c412596907543f9ad62d3, 70d6f05162774770ac0d62fc64d62e57; no voters:
I20251026 15:08:07.367802 20438 raft_consensus.cc:2804] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:07.367985 20438 raft_consensus.cc:697] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [term 1 LEADER]: Becoming Leader. State: Replica: 70d6f05162774770ac0d62fc64d62e57, State: Running, Role: LEADER
I20251026 15:08:07.368070 20438 consensus_queue.cc:237] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053323e7f8c412596907543f9ad62d3" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36643 } } peers { permanent_uuid: "97ecb53af2ef49379a8bbf3fb84aa471" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43985 } } peers { permanent_uuid: "70d6f05162774770ac0d62fc64d62e57" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44003 } }
W20251026 15:08:07.368512 20415 tablet.cc:2378] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:08:07.368839 19949 catalog_manager.cc:5649] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70d6f05162774770ac0d62fc64d62e57 (127.14.212.66). New cstate: current_term: 1 leader_uuid: "70d6f05162774770ac0d62fc64d62e57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053323e7f8c412596907543f9ad62d3" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ecb53af2ef49379a8bbf3fb84aa471" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70d6f05162774770ac0d62fc64d62e57" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 44003 } health_report { overall_health: HEALTHY } } }
I20251026 15:08:07.379549 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.377635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.14.212.66:44003)
1026 15:08:07.377699 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.377701 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.377711 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:07.377748 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.377748 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.377765 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.377818 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:08:07.378041 (+ 223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.378563 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:07.379094 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:07.379216 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:07.379248 (+ 32us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:08:07.379341 (+ 93us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:08:07.379402 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.379448 (+ 46us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.379465 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251026 15:08:07.381179 20096 raft_consensus.cc:1275] T 21580042a60f484490005a8f24260a89 P 97ecb53af2ef49379a8bbf3fb84aa471 [term 1 FOLLOWER]: Refusing update from remote peer 70d6f05162774770ac0d62fc64d62e57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:07.381198 20368 raft_consensus.cc:1275] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3 [term 1 FOLLOWER]: Refusing update from remote peer 70d6f05162774770ac0d62fc64d62e57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:07.381492 20438 consensus_queue.cc:1048] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2053323e7f8c412596907543f9ad62d3" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:07.381618 20441 consensus_queue.cc:1048] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ecb53af2ef49379a8bbf3fb84aa471" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:07.383860 20444 mvcc.cc:204] Tried to move back new op lower bound from 7215068313110917120 to 7215068313060220928. Current Snapshot: MvccSnapshot[applied={T|T < 7215068313110917120}]
I20251026 15:08:07.384009 20445 mvcc.cc:204] Tried to move back new op lower bound from 7215068313110917120 to 7215068313060220928. Current Snapshot: MvccSnapshot[applied={T|T < 7215068313110917120}]
I20251026 15:08:07.384166 20446 mvcc.cc:204] Tried to move back new op lower bound from 7215068313110917120 to 7215068313060220928. Current Snapshot: MvccSnapshot[applied={T|T < 7215068313110917120}]
I20251026 15:08:07.386145 19949 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41798:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:07.386250 19949 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41798:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:07.386957 19949 catalog_manager.cc:5936] T 00000000000000000000000000000000 P cfb15d291c2d4121992bf44b3378c0c5: Sending DeleteTablet for 3 replicas of tablet 21580042a60f484490005a8f24260a89
I20251026 15:08:07.387271 20348 tablet_service.cc:1552] Processing DeleteTablet for tablet 21580042a60f484490005a8f24260a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:07 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:45944
I20251026 15:08:07.387312 20076 tablet_service.cc:1552] Processing DeleteTablet for tablet 21580042a60f484490005a8f24260a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:07 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:37258
I20251026 15:08:07.387312 20212 tablet_service.cc:1552] Processing DeleteTablet for tablet 21580042a60f484490005a8f24260a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:07 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:48460
I20251026 15:08:07.387594 20455 tablet_replica.cc:333] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57: stopping tablet replica
I20251026 15:08:07.387681 20455 raft_consensus.cc:2243] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:08:07.387785 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 20014
I20251026 15:08:07.390650 20455 raft_consensus.cc:2272] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:07.390846 20454 tablet_replica.cc:333] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3: stopping tablet replica
I20251026 15:08:07.390938 20454 raft_consensus.cc:2243] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:07.391043 20454 raft_consensus.cc:2272] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:07.391222 20455 ts_tablet_manager.cc:1916] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:07.391474 20454 ts_tablet_manager.cc:1916] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:07.393302 20454 ts_tablet_manager.cc:1929] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:08:07.393410 20454 log.cc:1199] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/21580042a60f484490005a8f24260a89
I20251026 15:08:07.393553 20455 ts_tablet_manager.cc:1929] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:08:07.393682 20455 log.cc:1199] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/21580042a60f484490005a8f24260a89
I20251026 15:08:07.393738 20454 ts_tablet_manager.cc:1950] T 21580042a60f484490005a8f24260a89 P 2053323e7f8c412596907543f9ad62d3: Deleting consensus metadata
I20251026 15:08:07.394021 20455 ts_tablet_manager.cc:1950] T 21580042a60f484490005a8f24260a89 P 70d6f05162774770ac0d62fc64d62e57: Deleting consensus metadata
I20251026 15:08:07.394212 19935 catalog_manager.cc:4985] TS 2053323e7f8c412596907543f9ad62d3 (127.14.212.67:36643): tablet 21580042a60f484490005a8f24260a89 (table test-table [id=b97fb750a8db4850abe85c435cec9ee1]) successfully deleted
I20251026 15:08:07.394691 19934 catalog_manager.cc:4985] TS 70d6f05162774770ac0d62fc64d62e57 (127.14.212.66:44003): tablet 21580042a60f484490005a8f24260a89 (table test-table [id=b97fb750a8db4850abe85c435cec9ee1]) successfully deleted
W20251026 15:08:07.395207 19935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:43985 (error 108)
I20251026 15:08:07.395434 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 20150
I20251026 15:08:07.397179 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.397003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:43985 (local address 127.0.0.1:37264)
1026 15:08:07.397078 (+ 75us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:07.397128 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20251026 15:08:07.397387 19935 catalog_manager.cc:4712] TS 97ecb53af2ef49379a8bbf3fb84aa471 (127.14.212.65:43985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21580042a60f484490005a8f24260a89: Network error: Client connection negotiation failed: client connection to 127.14.212.65:43985: connect: Connection refused (error 111)
I20251026 15:08:07.400633 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 20286
I20251026 15:08:07.405622 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 19918
2025-10-26T15:08:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3064 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-10-26T15:08:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:07Z Disabled control of system clock
I20251026 15:08:07.432723 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33299
--webserver_interface=127.14.212.126
--webserver_port=0
--builtin_ntp_servers=127.14.212.84:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:33299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251026 15:08:07.512398 20461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:07.512585 20461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:07.512617 20461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:07.514108 20461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:07.514164 20461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:07.514184 20461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:07.514206 20461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:07.514225 20461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:08:07.515846 20461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:33299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:07.516106 20461 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:07.516343 20461 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:07.519086 20469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:07.519099 20467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:07.519198 20466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:07.519387 20461 server_base.cc:1047] running on GCE node
I20251026 15:08:07.519605 20461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:07.519855 20461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:07.520985 20461 hybrid_clock.cc:648] HybridClock initialized: now 1761491287520963 us; error 34 us; skew 500 ppm
I20251026 15:08:07.522192 20461 webserver.cc:492] Webserver started at http://127.14.212.126:34583/ using document root <none> and password file <none>
I20251026 15:08:07.522426 20461 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:07.522485 20461 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:07.522604 20461 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:07.523576 20461 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "c2bf38242a39494d8044b0ba291d8bd3"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "a5862975685dd94c1ee46d40dc55b28e"
server_key_iv: "0918e5f17983cbc0120fe01e9704c1c3"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.523893 20461 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "c2bf38242a39494d8044b0ba291d8bd3"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "a5862975685dd94c1ee46d40dc55b28e"
server_key_iv: "0918e5f17983cbc0120fe01e9704c1c3"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.525125 20461 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:08:07.525903 20475 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:07.526091 20461 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:07.526181 20461 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "c2bf38242a39494d8044b0ba291d8bd3"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "a5862975685dd94c1ee46d40dc55b28e"
server_key_iv: "0918e5f17983cbc0120fe01e9704c1c3"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.526253 20461 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:07.529824 20461 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:07.530066 20461 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:07.530174 20461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:07.534050 20461 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:33299
I20251026 15:08:07.534081 20527 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:33299 every 8 connection(s)
I20251026 15:08:07.534475 20461 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:07.535142 20528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:07.536634 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 20461
I20251026 15:08:07.536739 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:08:07.536929 15185 external_mini_cluster.cc:1468] Setting key 8fac035f4277f36634ce476af67f98a4
I20251026 15:08:07.538013 20528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3: Bootstrap starting.
I20251026 15:08:07.538646 20528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:07.538951 20528 log.cc:826] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:07.539747 20528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3: No bootstrap required, opened a new log
I20251026 15:08:07.540508 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.537483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.14.212.126:33299)
1026 15:08:07.537701 (+ 218us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.537711 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.538608 (+ 897us) server_negotiation.cc:408] Connection header received
1026 15:08:07.539043 (+ 435us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.539051 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.539112 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.539375 (+ 263us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:07.540107 (+ 732us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:07.540111 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:07.540111 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:07.540115 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:07.540146 (+ 31us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:07.540153 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:07.540224 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.540293 (+ 69us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.540336 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20251026 15:08:07.541217 20528 raft_consensus.cc:359] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bf38242a39494d8044b0ba291d8bd3" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33299 } }
I20251026 15:08:07.541342 20528 raft_consensus.cc:385] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:07.541373 20528 raft_consensus.cc:740] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2bf38242a39494d8044b0ba291d8bd3, State: Initialized, Role: FOLLOWER
I20251026 15:08:07.541481 20528 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bf38242a39494d8044b0ba291d8bd3" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33299 } }
I20251026 15:08:07.541549 20528 raft_consensus.cc:399] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:07.541585 20528 raft_consensus.cc:493] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:07.541632 20528 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:07.542174 20528 raft_consensus.cc:515] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bf38242a39494d8044b0ba291d8bd3" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33299 } }
I20251026 15:08:07.542277 20528 leader_election.cc:304] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2bf38242a39494d8044b0ba291d8bd3; no voters:
I20251026 15:08:07.542552 20528 leader_election.cc:290] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:08:07.542620 20533 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:07.542771 20533 raft_consensus.cc:697] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [term 1 LEADER]: Becoming Leader. State: Replica: c2bf38242a39494d8044b0ba291d8bd3, State: Running, Role: LEADER
I20251026 15:08:07.542801 20528 sys_catalog.cc:565] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:07.542855 20533 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bf38242a39494d8044b0ba291d8bd3" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33299 } }
I20251026 15:08:07.543200 20533 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2bf38242a39494d8044b0ba291d8bd3. Latest consensus state: current_term: 1 leader_uuid: "c2bf38242a39494d8044b0ba291d8bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bf38242a39494d8044b0ba291d8bd3" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33299 } } }
I20251026 15:08:07.543269 20533 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:07.543226 20535 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2bf38242a39494d8044b0ba291d8bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bf38242a39494d8044b0ba291d8bd3" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33299 } } }
I20251026 15:08:07.543357 20535 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:07.543538 20537 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:07.544011 20537 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:08:07.545861 20537 catalog_manager.cc:1357] Generated new cluster ID: 8ece8471d3364601880ba14df670ef27
I20251026 15:08:07.545907 20537 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:08:07.559106 20537 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:08:07.559275 20537 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:07.562307 20537 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3: Generated new TSK 0
I20251026 15:08:07.562536 20537 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:08:07.564064 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33299
--builtin_ntp_servers=127.14.212.84:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251026 15:08:07.651440 20552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:07.651638 20552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:07.651677 20552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:07.653262 20552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:07.653398 20552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:07.653512 20552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:08:07.655441 20552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:07.655740 20552 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:07.656015 20552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:07.659046 20558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:07.659034 20560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:07.659034 20557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:07.659377 20552 server_base.cc:1047] running on GCE node
I20251026 15:08:07.659572 20552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:07.659827 20552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:07.661026 20552 hybrid_clock.cc:648] HybridClock initialized: now 1761491287661010 us; error 37 us; skew 500 ppm
I20251026 15:08:07.662600 20552 webserver.cc:492] Webserver started at http://127.14.212.65:42587/ using document root <none> and password file <none>
I20251026 15:08:07.662850 20552 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:07.662914 20552 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:07.663038 20552 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:07.664350 20552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "b42ce56585bc4037901fc9188ffca7c8"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "929878ec82f2e73e1b9a4e63fef7a38a"
server_key_iv: "acf16b61ff2e99fe96bd5f93d8dfb4c2"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.664727 20552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "b42ce56585bc4037901fc9188ffca7c8"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "929878ec82f2e73e1b9a4e63fef7a38a"
server_key_iv: "acf16b61ff2e99fe96bd5f93d8dfb4c2"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.666857 20552 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 15:08:07.668246 20566 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:07.668437 20552 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:07.668532 20552 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "b42ce56585bc4037901fc9188ffca7c8"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "929878ec82f2e73e1b9a4e63fef7a38a"
server_key_iv: "acf16b61ff2e99fe96bd5f93d8dfb4c2"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.668601 20552 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:07.673871 20552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:07.674146 20552 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:07.674263 20552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:07.674497 20552 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:07.674841 20552 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:07.674889 20552 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:07.674921 20552 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:07.674939 20552 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:07.682103 20552 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:39027
I20251026 15:08:07.682165 20679 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:39027 every 8 connection(s)
I20251026 15:08:07.682581 20552 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:08:07.685640 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.683422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:56827 (local address 127.14.212.126:33299)
1026 15:08:07.683619 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.683621 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.683671 (+ 50us) server_negotiation.cc:408] Connection header received
1026 15:08:07.684182 (+ 511us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.684183 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.684189 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.684265 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:07.685309 (+ 1044us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:07.685311 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:07.685311 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:07.685316 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:07.685337 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:07.685341 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:07.685407 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.685546 (+ 139us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.685564 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20251026 15:08:07.685791 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.683408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33299 (local address 127.14.212.65:56827)
1026 15:08:07.683582 (+ 174us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:07.683608 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:07.684066 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:07.684430 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:07.684433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:07.684450 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:07.685160 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:08:07.685161 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:07.685190 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:08:07.685211 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:07.685444 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:07.685450 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:07.685454 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:07.685563 (+ 109us) client_negotiation.cc:240] Negotiation successful
1026 15:08:07.685610 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20251026 15:08:07.686605 20680 heartbeater.cc:344] Connected to a master server at 127.14.212.126:33299
I20251026 15:08:07.686708 20680 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:07.686851 20680 heartbeater.cc:507] Master 127.14.212.126:33299 requested a full tablet report, sending...
I20251026 15:08:07.687337 20492 ts_manager.cc:194] Registered new tserver with Master: b42ce56585bc4037901fc9188ffca7c8 (127.14.212.65:39027)
I20251026 15:08:07.689412 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 20552
I20251026 15:08:07.689527 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:08:07.689711 15185 external_mini_cluster.cc:1468] Setting key b8b252c6a8d8cd1431b06449d4dd89a0
I20251026 15:08:07.690761 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33299
--builtin_ntp_servers=127.14.212.84:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251026 15:08:07.772607 20683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:07.772796 20683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:07.772818 20683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:07.774341 20683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:07.774385 20683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:07.774449 20683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:08:07.776043 20683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:07.776275 20683 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:07.776500 20683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:07.778990 20688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:07.778995 20689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:07.779139 20683 server_base.cc:1047] running on GCE node
W20251026 15:08:07.778996 20691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:07.779399 20683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:07.779608 20683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:07.780743 20683 hybrid_clock.cc:648] HybridClock initialized: now 1761491287780726 us; error 37 us; skew 500 ppm
I20251026 15:08:07.781924 20683 webserver.cc:492] Webserver started at http://127.14.212.66:40431/ using document root <none> and password file <none>
I20251026 15:08:07.782141 20683 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:07.782189 20683 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:07.782303 20683 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:07.783306 20683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "bfadd7a865ec452a967fcc6ac60fcdb7"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "2ae89437d00daa63e306fc3898079f92"
server_key_iv: "4fd99a9b53f570af6172717c7d3f7318"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.783629 20683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "bfadd7a865ec452a967fcc6ac60fcdb7"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "2ae89437d00daa63e306fc3898079f92"
server_key_iv: "4fd99a9b53f570af6172717c7d3f7318"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.784871 20683 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:08:07.785622 20697 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:07.785807 20683 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:08:07.785879 20683 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "bfadd7a865ec452a967fcc6ac60fcdb7"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "2ae89437d00daa63e306fc3898079f92"
server_key_iv: "4fd99a9b53f570af6172717c7d3f7318"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.785946 20683 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:07.789436 20683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:07.789659 20683 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:07.789765 20683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:07.789942 20683 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:07.790226 20683 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:07.790261 20683 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:07.790293 20683 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:07.790323 20683 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:07.796068 20683 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:37837
I20251026 15:08:07.796200 20810 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:37837 every 8 connection(s)
I20251026 15:08:07.796437 20683 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:08:07.799513 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.797217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:53745 (local address 127.14.212.126:33299)
1026 15:08:07.797296 (+ 79us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.797299 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.797493 (+ 194us) server_negotiation.cc:408] Connection header received
1026 15:08:07.798066 (+ 573us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.798068 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.798076 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.798153 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:07.798979 (+ 826us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:07.798980 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:07.798981 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:07.798986 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:07.799093 (+ 107us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:07.799111 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:07.799211 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.799410 (+ 199us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.799434 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251026 15:08:07.799710 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.797146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33299 (local address 127.14.212.66:53745)
1026 15:08:07.797390 (+ 244us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:07.797415 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:07.797958 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:07.798225 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:07.798229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:07.798238 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:07.798833 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:08:07.798833 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:07.798867 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:08:07.798890 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:07.799227 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:07.799233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:07.799237 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:07.799436 (+ 199us) client_negotiation.cc:240] Negotiation successful
1026 15:08:07.799496 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20251026 15:08:07.800200 20811 heartbeater.cc:344] Connected to a master server at 127.14.212.126:33299
I20251026 15:08:07.800298 20811 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:07.800413 20811 heartbeater.cc:507] Master 127.14.212.126:33299 requested a full tablet report, sending...
I20251026 15:08:07.800769 20492 ts_manager.cc:194] Registered new tserver with Master: bfadd7a865ec452a967fcc6ac60fcdb7 (127.14.212.66:37837)
I20251026 15:08:07.805394 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 20683
I20251026 15:08:07.805485 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:08:07.805634 15185 external_mini_cluster.cc:1468] Setting key 00c2be1dfa278049c92cd612b22db5b8
I20251026 15:08:07.806813 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33299
--builtin_ntp_servers=127.14.212.84:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251026 15:08:07.886973 20814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:07.887161 20814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:07.887182 20814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:07.888572 20814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:07.888614 20814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:07.888638 20814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:08:07.890254 20814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:07.890489 20814 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:07.890734 20814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:08:07.893463 20814 server_base.cc:1047] running on GCE node
W20251026 15:08:07.893440 20820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:07.893437 20819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:07.893437 20822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:07.893780 20814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:07.893991 20814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:07.895139 20814 hybrid_clock.cc:648] HybridClock initialized: now 1761491287895123 us; error 27 us; skew 500 ppm
I20251026 15:08:07.896302 20814 webserver.cc:492] Webserver started at http://127.14.212.67:36019/ using document root <none> and password file <none>
I20251026 15:08:07.896497 20814 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:07.896546 20814 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:07.896674 20814 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:07.897641 20814 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "d6044f4822504d9397fc62b8f1daad7c"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "5f56b25557865c6917bf65d8932d7f0e"
server_key_iv: "022246f4a12529e1e6573e4cbbdcd3a2"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.897975 20814 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "d6044f4822504d9397fc62b8f1daad7c"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "5f56b25557865c6917bf65d8932d7f0e"
server_key_iv: "022246f4a12529e1e6573e4cbbdcd3a2"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.899354 20814 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:08:07.900282 20829 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:07.900527 20814 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:07.900611 20814 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "d6044f4822504d9397fc62b8f1daad7c"
format_stamp: "Formatted at 2025-10-26 15:08:07 on dist-test-slave-8pgz"
server_key: "5f56b25557865c6917bf65d8932d7f0e"
server_key_iv: "022246f4a12529e1e6573e4cbbdcd3a2"
server_key_version: "encryptionkey@0"
I20251026 15:08:07.900681 20814 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:07.904493 20814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:07.904695 20814 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:07.904778 20814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:07.904917 20814 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:07.905201 20814 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:07.905234 20814 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:07.905256 20814 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:07.905290 20814 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:07.910966 20814 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:44595
I20251026 15:08:07.911036 20942 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:44595 every 8 connection(s)
I20251026 15:08:07.911339 20814 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:08:07.913992 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.912124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:60435 (local address 127.14.212.126:33299)
1026 15:08:07.912233 (+ 109us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.912236 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.912387 (+ 151us) server_negotiation.cc:408] Connection header received
1026 15:08:07.912870 (+ 483us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.912872 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.912878 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.912942 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:07.913732 (+ 790us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:07.913734 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:07.913734 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:07.913741 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:07.913756 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:07.913760 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:07.913812 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.913894 (+ 82us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.913911 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20251026 15:08:07.914201 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.912015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33299 (local address 127.14.212.67:60435)
1026 15:08:07.912280 (+ 265us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:07.912306 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:07.912779 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:07.913018 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:07.913021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:07.913031 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:07.913600 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:08:07.913600 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:07.913623 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:08:07.913642 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:07.913815 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:07.913819 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:07.913822 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:08:07.913946 (+ 124us) client_negotiation.cc:240] Negotiation successful
1026 15:08:07.914004 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":85,"threads_started":1}
I20251026 15:08:07.914697 20943 heartbeater.cc:344] Connected to a master server at 127.14.212.126:33299
I20251026 15:08:07.914783 20943 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:07.914906 20943 heartbeater.cc:507] Master 127.14.212.126:33299 requested a full tablet report, sending...
I20251026 15:08:07.915295 20492 ts_manager.cc:194] Registered new tserver with Master: d6044f4822504d9397fc62b8f1daad7c (127.14.212.67:44595)
I20251026 15:08:07.921221 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 20814
I20251026 15:08:07.921314 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:08:07.921482 15185 external_mini_cluster.cc:1468] Setting key 757c987f7dac76433d954ff2b9075524
I20251026 15:08:07.922665 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:08:07.925073 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.924041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.14.212.126:33299)
1026 15:08:07.924098 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.924100 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.924184 (+ 84us) server_negotiation.cc:408] Connection header received
1026 15:08:07.924240 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.924241 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.924246 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.924297 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:07.924896 (+ 599us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:07.924897 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:07.924897 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:07.924899 (+ 2us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:07.924913 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:07.924916 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:07.924952 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.925006 (+ 54us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.925019 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20251026 15:08:07.926623 20492 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:08:07.928059 20492 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:08:07.932107 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.930561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:37837 (local address 127.0.0.1:48192)
1026 15:08:07.930679 (+ 118us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:07.930688 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:07.930728 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:07.930887 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:07.930892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:07.930897 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:07.931520 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:08:07.931521 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:07.931552 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:08:07.931645 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:07.931937 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:07.931942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:07.931946 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:07.932009 (+ 63us) client_negotiation.cc:240] Negotiation successful
1026 15:08:07.932025 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20251026 15:08:07.932384 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.930281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:39027 (local address 127.0.0.1:43970)
1026 15:08:07.930593 (+ 312us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:07.930605 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:07.930653 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:07.931440 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:07.931441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:07.931444 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:07.931889 (+ 445us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:08:07.931889 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:07.931898 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:08:07.931911 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:07.932259 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:07.932260 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:07.932261 (+ 1us) client_negotiation.cc:769] Sending connection context
1026 15:08:07.932303 (+ 42us) client_negotiation.cc:240] Negotiation successful
1026 15:08:07.932316 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":88,"threads_started":1}
I20251026 15:08:07.932546 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.930585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.14.212.66:37837)
1026 15:08:07.930744 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.930753 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.930765 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:08:07.930791 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.930796 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.930803 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.930865 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:07.931741 (+ 876us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:07.931750 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:07.931750 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:07.931755 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:07.931810 (+ 55us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:07.931819 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:07.931896 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.932219 (+ 323us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.932235 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20251026 15:08:07.932801 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.930213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:44595 (local address 127.0.0.1:44028)
1026 15:08:07.930483 (+ 270us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:07.930506 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:07.930570 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:07.930930 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:07.930931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:07.930958 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:07.932428 (+ 1470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:08:07.932428 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:07.932438 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:08:07.932450 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:07.932690 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:07.932691 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:07.932692 (+ 1us) client_negotiation.cc:769] Sending connection context
1026 15:08:07.932718 (+ 26us) client_negotiation.cc:240] Negotiation successful
1026 15:08:07.932732 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":83,"threads_started":1}
I20251026 15:08:07.932826 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.930421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.14.212.65:39027)
1026 15:08:07.931299 (+ 878us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.931306 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.931313 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:08:07.931335 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.931337 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.931345 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.931419 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:07.932114 (+ 695us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:07.932117 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:07.932118 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:07.932121 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:07.932142 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:07.932148 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:07.932231 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.932568 (+ 337us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.932585 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":97,"threads_started":1}
I20251026 15:08:07.933138 20745 tablet_service.cc:1505] Processing CreateTablet for tablet 1c3ee664734644fea2185d77ed537093 (DEFAULT_TABLE table=test-table [id=40d918290eb540fe9ba586bcee3a53eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:07.933256 20614 tablet_service.cc:1505] Processing CreateTablet for tablet 1c3ee664734644fea2185d77ed537093 (DEFAULT_TABLE table=test-table [id=40d918290eb540fe9ba586bcee3a53eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:07.933343 20953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:07.930334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44028 (local address 127.14.212.67:44595)
1026 15:08:07.930738 (+ 404us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:07.930753 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:07.930769 (+ 16us) server_negotiation.cc:408] Connection header received
1026 15:08:07.930791 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:07.930793 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:07.930800 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:07.930914 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:07.932544 (+ 1630us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:07.932551 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:07.932552 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:07.932556 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:07.932605 (+ 49us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:07.932612 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:07.933009 (+ 397us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:07.933035 (+ 26us) server_negotiation.cc:300] Negotiation successful
1026 15:08:07.933049 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":99,"threads_started":1}
I20251026 15:08:07.933463 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3ee664734644fea2185d77ed537093. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:07.933476 20614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3ee664734644fea2185d77ed537093. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:07.933807 20877 tablet_service.cc:1505] Processing CreateTablet for tablet 1c3ee664734644fea2185d77ed537093 (DEFAULT_TABLE table=test-table [id=40d918290eb540fe9ba586bcee3a53eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:07.934054 20877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3ee664734644fea2185d77ed537093. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:07.936369 20957 tablet_bootstrap.cc:492] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c: Bootstrap starting.
I20251026 15:08:07.937330 20957 tablet_bootstrap.cc:654] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:07.937592 20957 log.cc:826] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:07.938301 20957 tablet_bootstrap.cc:492] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c: No bootstrap required, opened a new log
I20251026 15:08:07.938366 20957 ts_tablet_manager.cc:1403] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:08:07.939534 20957 raft_consensus.cc:359] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6044f4822504d9397fc62b8f1daad7c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 44595 } } peers { permanent_uuid: "b42ce56585bc4037901fc9188ffca7c8" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 39027 } } peers { permanent_uuid: "bfadd7a865ec452a967fcc6ac60fcdb7" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 37837 } }
I20251026 15:08:07.939663 20957 raft_consensus.cc:385] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:07.939697 20957 raft_consensus.cc:740] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6044f4822504d9397fc62b8f1daad7c, State: Initialized, Role: FOLLOWER
I20251026 15:08:07.939805 20957 consensus_queue.cc:260] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6044f4822504d9397fc62b8f1daad7c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 44595 } } peers { permanent_uuid: "b42ce56585bc4037901fc9188ffca7c8" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 39027 } } peers { permanent_uuid: "bfadd7a865ec452a967fcc6ac60fcdb7" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 37837 } }
I20251026 15:08:07.940116 20957 ts_tablet_manager.cc:1434] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:08:07.940240 20943 heartbeater.cc:499] Master 127.14.212.126:33299 was elected leader, sending a full tablet report...
I20251026 15:08:07.941483 20958 tablet_bootstrap.cc:492] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8: Bootstrap starting.
I20251026 15:08:07.941718 20961 tablet_bootstrap.cc:492] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7: Bootstrap starting.
I20251026 15:08:07.942217 20958 tablet_bootstrap.cc:654] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:07.942355 20961 tablet_bootstrap.cc:654] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:07.942566 20958 log.cc:826] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:07.942682 20961 log.cc:826] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:07.943434 20961 tablet_bootstrap.cc:492] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7: No bootstrap required, opened a new log
I20251026 15:08:07.943506 20961 ts_tablet_manager.cc:1403] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 15:08:07.944190 20958 tablet_bootstrap.cc:492] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8: No bootstrap required, opened a new log
I20251026 15:08:07.944262 20958 ts_tablet_manager.cc:1403] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251026 15:08:07.945291 20961 raft_consensus.cc:359] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6044f4822504d9397fc62b8f1daad7c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 44595 } } peers { permanent_uuid: "b42ce56585bc4037901fc9188ffca7c8" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 39027 } } peers { permanent_uuid: "bfadd7a865ec452a967fcc6ac60fcdb7" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 37837 } }
I20251026 15:08:07.945453 20961 raft_consensus.cc:385] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:07.945488 20961 raft_consensus.cc:740] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfadd7a865ec452a967fcc6ac60fcdb7, State: Initialized, Role: FOLLOWER
I20251026 15:08:07.945586 20961 consensus_queue.cc:260] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6044f4822504d9397fc62b8f1daad7c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 44595 } } peers { permanent_uuid: "b42ce56585bc4037901fc9188ffca7c8" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 39027 } } peers { permanent_uuid: "bfadd7a865ec452a967fcc6ac60fcdb7" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 37837 } }
I20251026 15:08:07.945849 20961 ts_tablet_manager.cc:1434] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251026 15:08:07.945884 20958 raft_consensus.cc:359] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6044f4822504d9397fc62b8f1daad7c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 44595 } } peers { permanent_uuid: "b42ce56585bc4037901fc9188ffca7c8" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 39027 } } peers { permanent_uuid: "bfadd7a865ec452a967fcc6ac60fcdb7" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 37837 } }
I20251026 15:08:07.946031 20958 raft_consensus.cc:385] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:07.946069 20958 raft_consensus.cc:740] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b42ce56585bc4037901fc9188ffca7c8, State: Initialized, Role: FOLLOWER
I20251026 15:08:07.946045 20811 heartbeater.cc:499] Master 127.14.212.126:33299 was elected leader, sending a full tablet report...
I20251026 15:08:07.946141 20958 consensus_queue.cc:260] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6044f4822504d9397fc62b8f1daad7c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 44595 } } peers { permanent_uuid: "b42ce56585bc4037901fc9188ffca7c8" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 39027 } } peers { permanent_uuid: "bfadd7a865ec452a967fcc6ac60fcdb7" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 37837 } }
I20251026 15:08:07.946377 20958 ts_tablet_manager.cc:1434] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:08:07.946432 20680 heartbeater.cc:499] Master 127.14.212.126:33299 was elected leader, sending a full tablet report...
I20251026 15:08:08.038976 20960 raft_consensus.cc:493] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:08:08.039103 20960 raft_consensus.cc:515] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6044f4822504d9397fc62b8f1daad7c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 44595 } } peers { permanent_uuid: "b42ce56585bc4037901fc9188ffca7c8" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 39027 } } peers { permanent_uuid: "bfadd7a865ec452a967fcc6ac60fcdb7" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 37837 } }
I20251026 15:08:08.039422 20960 leader_election.cc:290] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b42ce56585bc4037901fc9188ffca7c8 (127.14.212.65:39027), bfadd7a865ec452a967fcc6ac60fcdb7 (127.14.212.66:37837)
I20251026 15:08:08.040901 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:08.039755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:40629 (local address 127.14.212.66:37837)
1026 15:08:08.039793 (+ 38us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:08.039796 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:08.039806 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:08.039840 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:08.039841 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:08.039848 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:08.039920 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:08.040583 (+ 663us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:08.040585 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:08.040585 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:08.040590 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:08.040610 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:08.040614 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:08.040672 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:08.040801 (+ 129us) server_negotiation.cc:300] Negotiation successful
1026 15:08:08.040819 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251026 15:08:08.040891 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:08.039624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:37837 (local address 127.14.212.67:40629)
1026 15:08:08.039710 (+ 86us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:08.039725 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:08.039770 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:08.039951 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:08.039953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:08.039955 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:08.040465 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:08:08.040466 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:08.040474 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:08:08.040489 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:08.040718 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:08.040720 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:08.040721 (+ 1us) client_negotiation.cc:769] Sending connection context
1026 15:08:08.040803 (+ 82us) client_negotiation.cc:240] Negotiation successful
1026 15:08:08.040819 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20251026 15:08:08.041249 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:08.039694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:53699 (local address 127.14.212.65:39027)
1026 15:08:08.039736 (+ 42us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:08.039738 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:08.039935 (+ 197us) server_negotiation.cc:408] Connection header received
1026 15:08:08.039965 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:08.039967 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:08.039974 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:08.040039 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:08.040971 (+ 932us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:08.040972 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:08.040973 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:08.040978 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:08.040996 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:08.041001 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:08.041061 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:08.041155 (+ 94us) server_negotiation.cc:300] Negotiation successful
1026 15:08:08.041179 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251026 15:08:08.041270 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:08.039626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:39027 (local address 127.14.212.67:53699)
1026 15:08:08.039807 (+ 181us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:08.039819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:08.039886 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:08.040061 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:08.040062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:08.040063 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:08.040751 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:08:08.040751 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:08.040759 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:08:08.040772 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:08.041062 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:08.041063 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:08.041064 (+ 1us) client_negotiation.cc:769] Sending connection context
1026 15:08:08.041140 (+ 76us) client_negotiation.cc:240] Negotiation successful
1026 15:08:08.041172 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":11,"thread_start_us":115,"threads_started":1}
I20251026 15:08:08.041283 20765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c3ee664734644fea2185d77ed537093" candidate_uuid: "d6044f4822504d9397fc62b8f1daad7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfadd7a865ec452a967fcc6ac60fcdb7" is_pre_election: true
I20251026 15:08:08.041471 20765 raft_consensus.cc:2468] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6044f4822504d9397fc62b8f1daad7c in term 0.
I20251026 15:08:08.041653 20620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c3ee664734644fea2185d77ed537093" candidate_uuid: "d6044f4822504d9397fc62b8f1daad7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b42ce56585bc4037901fc9188ffca7c8" is_pre_election: true
I20251026 15:08:08.041829 20620 raft_consensus.cc:2468] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6044f4822504d9397fc62b8f1daad7c in term 0.
I20251026 15:08:08.042030 20830 leader_election.cc:304] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfadd7a865ec452a967fcc6ac60fcdb7, d6044f4822504d9397fc62b8f1daad7c; no voters:
I20251026 15:08:08.042167 20960 raft_consensus.cc:2804] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:08:08.042233 20960 raft_consensus.cc:493] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:08:08.042264 20960 raft_consensus.cc:3060] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:08.043004 20960 raft_consensus.cc:515] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6044f4822504d9397fc62b8f1daad7c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 44595 } } peers { permanent_uuid: "b42ce56585bc4037901fc9188ffca7c8" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 39027 } } peers { permanent_uuid: "bfadd7a865ec452a967fcc6ac60fcdb7" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 37837 } }
I20251026 15:08:08.043128 20960 leader_election.cc:290] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [CANDIDATE]: Term 1 election: Requested vote from peers b42ce56585bc4037901fc9188ffca7c8 (127.14.212.65:39027), bfadd7a865ec452a967fcc6ac60fcdb7 (127.14.212.66:37837)
I20251026 15:08:08.043321 20620 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c3ee664734644fea2185d77ed537093" candidate_uuid: "d6044f4822504d9397fc62b8f1daad7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b42ce56585bc4037901fc9188ffca7c8"
I20251026 15:08:08.043339 20765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c3ee664734644fea2185d77ed537093" candidate_uuid: "d6044f4822504d9397fc62b8f1daad7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfadd7a865ec452a967fcc6ac60fcdb7"
I20251026 15:08:08.043402 20620 raft_consensus.cc:3060] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:08.043426 20765 raft_consensus.cc:3060] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:08.044428 20765 raft_consensus.cc:2468] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6044f4822504d9397fc62b8f1daad7c in term 1.
I20251026 15:08:08.044517 20620 raft_consensus.cc:2468] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6044f4822504d9397fc62b8f1daad7c in term 1.
I20251026 15:08:08.044596 20830 leader_election.cc:304] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfadd7a865ec452a967fcc6ac60fcdb7, d6044f4822504d9397fc62b8f1daad7c; no voters:
I20251026 15:08:08.044684 20960 raft_consensus.cc:2804] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:08.044832 20960 raft_consensus.cc:697] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [term 1 LEADER]: Becoming Leader. State: Replica: d6044f4822504d9397fc62b8f1daad7c, State: Running, Role: LEADER
I20251026 15:08:08.044936 20960 consensus_queue.cc:237] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6044f4822504d9397fc62b8f1daad7c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 44595 } } peers { permanent_uuid: "b42ce56585bc4037901fc9188ffca7c8" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 39027 } } peers { permanent_uuid: "bfadd7a865ec452a967fcc6ac60fcdb7" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 37837 } }
I20251026 15:08:08.045761 20492 catalog_manager.cc:5649] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c reported cstate change: term changed from 0 to 1, leader changed from <none> to d6044f4822504d9397fc62b8f1daad7c (127.14.212.67). New cstate: current_term: 1 leader_uuid: "d6044f4822504d9397fc62b8f1daad7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6044f4822504d9397fc62b8f1daad7c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 44595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b42ce56585bc4037901fc9188ffca7c8" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 39027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfadd7a865ec452a967fcc6ac60fcdb7" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 37837 } health_report { overall_health: UNKNOWN } } }
W20251026 15:08:08.046918 20812 tablet.cc:2378] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:08:08.051285 20953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:08.050359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.14.212.67:44595)
1026 15:08:08.050435 (+ 76us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:08.050437 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:08.050472 (+ 35us) server_negotiation.cc:408] Connection header received
1026 15:08:08.050504 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:08.050505 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:08.050515 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:08.050558 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:08.051133 (+ 575us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:08.051134 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:08.051134 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:08.051136 (+ 2us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:08.051146 (+ 10us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:08.051150 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:08.051186 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:08.051224 (+ 38us) server_negotiation.cc:300] Negotiation successful
1026 15:08:08.051235 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251026 15:08:08.052789 20765 raft_consensus.cc:1275] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7 [term 1 FOLLOWER]: Refusing update from remote peer d6044f4822504d9397fc62b8f1daad7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:08.052790 20620 raft_consensus.cc:1275] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8 [term 1 FOLLOWER]: Refusing update from remote peer d6044f4822504d9397fc62b8f1daad7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:08.053107 20960 consensus_queue.cc:1048] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfadd7a865ec452a967fcc6ac60fcdb7" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 37837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:08.053332 20967 consensus_queue.cc:1048] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b42ce56585bc4037901fc9188ffca7c8" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 39027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:08.055785 20971 mvcc.cc:204] Tried to move back new op lower bound from 7215068315861938176 to 7215068315832745984. Current Snapshot: MvccSnapshot[applied={T|T < 7215068315861938176}]
I20251026 15:08:08.055964 20972 mvcc.cc:204] Tried to move back new op lower bound from 7215068315861938176 to 7215068315832745984. Current Snapshot: MvccSnapshot[applied={T|T < 7215068315861938176}]
I20251026 15:08:08.058141 20491 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42034:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:08.058239 20491 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42034:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:08.059113 20491 catalog_manager.cc:5936] T 00000000000000000000000000000000 P c2bf38242a39494d8044b0ba291d8bd3: Sending DeleteTablet for 3 replicas of tablet 1c3ee664734644fea2185d77ed537093
I20251026 15:08:08.059444 20614 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c3ee664734644fea2185d77ed537093 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:08 UTC) from {username='slave'} at 127.0.0.1:43970
I20251026 15:08:08.059469 20745 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c3ee664734644fea2185d77ed537093 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:08 UTC) from {username='slave'} at 127.0.0.1:48192
I20251026 15:08:08.059728 20877 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c3ee664734644fea2185d77ed537093 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:08 UTC) from {username='slave'} at 127.0.0.1:44028
I20251026 15:08:08.059899 20981 tablet_replica.cc:333] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7: stopping tablet replica
I20251026 15:08:08.060014 20981 raft_consensus.cc:2243] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:08.060046 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 20552
I20251026 15:08:08.060014 20982 tablet_replica.cc:333] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c: stopping tablet replica
I20251026 15:08:08.060123 20981 raft_consensus.cc:2272] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:08.060241 20980 tablet_replica.cc:333] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8: stopping tablet replica
I20251026 15:08:08.060312 20980 raft_consensus.cc:2243] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:08.060391 20980 raft_consensus.cc:2272] T 1c3ee664734644fea2185d77ed537093 P b42ce56585bc4037901fc9188ffca7c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:08.061429 20981 ts_tablet_manager.cc:1916] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:08.061987 20982 raft_consensus.cc:2243] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:08:08.062134 20982 raft_consensus.cc:2272] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:08.062592 20982 ts_tablet_manager.cc:1916] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:08.063656 20981 ts_tablet_manager.cc:1929] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:08:08.063753 20981 log.cc:1199] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/1c3ee664734644fea2185d77ed537093
I20251026 15:08:08.064038 20981 ts_tablet_manager.cc:1950] T 1c3ee664734644fea2185d77ed537093 P bfadd7a865ec452a967fcc6ac60fcdb7: Deleting consensus metadata
I20251026 15:08:08.064407 20982 ts_tablet_manager.cc:1929] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:08:08.064462 20476 catalog_manager.cc:4985] TS bfadd7a865ec452a967fcc6ac60fcdb7 (127.14.212.66:37837): tablet 1c3ee664734644fea2185d77ed537093 (table test-table [id=40d918290eb540fe9ba586bcee3a53eb]) successfully deleted
I20251026 15:08:08.064570 20982 log.cc:1199] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/1c3ee664734644fea2185d77ed537093
I20251026 15:08:08.064893 20982 ts_tablet_manager.cc:1950] T 1c3ee664734644fea2185d77ed537093 P d6044f4822504d9397fc62b8f1daad7c: Deleting consensus metadata
I20251026 15:08:08.065331 20476 catalog_manager.cc:4985] TS d6044f4822504d9397fc62b8f1daad7c (127.14.212.67:44595): tablet 1c3ee664734644fea2185d77ed537093 (table test-table [id=40d918290eb540fe9ba586bcee3a53eb]) successfully deleted
W20251026 15:08:08.065769 20478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:39027 (error 108)
I20251026 15:08:08.065966 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 20683
I20251026 15:08:08.066994 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:08.066846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:39027 (local address 127.0.0.1:43972)
1026 15:08:08.066913 (+ 67us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:08.066954 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:39027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20251026 15:08:08.067112 20478 catalog_manager.cc:4712] TS b42ce56585bc4037901fc9188ffca7c8 (127.14.212.65:39027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c3ee664734644fea2185d77ed537093: Network error: Client connection negotiation failed: client connection to 127.14.212.65:39027: connect: Connection refused (error 111)
I20251026 15:08:08.071223 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 20814
I20251026 15:08:08.076854 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 20461
2025-10-26T15:08:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (670 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20251026 15:08:08.097107 15185 mini_oidc.cc:150] Starting JWKS server
I20251026 15:08:08.098127 15185 webserver.cc:492] Webserver started at https://127.0.0.1:38761/ using document root <none> and password file <none>
I20251026 15:08:08.098204 15185 mini_oidc.cc:186] Starting OIDC Discovery server
I20251026 15:08:08.098322 15185 webserver.cc:492] Webserver started at http://127.0.0.1:39851/ using document root <none> and password file <none>
2025-10-26T15:08:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:08Z Disabled control of system clock
I20251026 15:08:08.105754 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:38201
--webserver_interface=127.14.212.126
--webserver_port=0
--builtin_ntp_servers=127.14.212.84:37751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:38201
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38761/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/testchainca.cert with env {}
W20251026 15:08:08.185819 20989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:08.186017 20989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:08.186054 20989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:08.187559 20989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:08.187613 20989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:08.187635 20989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:08.187654 20989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:08.187673 20989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251026 15:08:08.187695 20989 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251026 15:08:08.187722 20989 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38761/jwks.json
I20251026 15:08:08.189349 20989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:37751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:38201
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:38201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38761/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:08.189626 20989 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:08.189800 20989 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:08.192808 20995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:08.193006 20997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:08.193030 20994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:08.193122 20989 server_base.cc:1047] running on GCE node
I20251026 15:08:08.193360 20989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:08.193691 20989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:08.194878 20989 hybrid_clock.cc:648] HybridClock initialized: now 1761491288194835 us; error 57 us; skew 500 ppm
I20251026 15:08:08.196151 20989 webserver.cc:492] Webserver started at http://127.14.212.126:35245/ using document root <none> and password file <none>
I20251026 15:08:08.196374 20989 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:08.196422 20989 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:08.196547 20989 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:08.197643 20989 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "61475c236ea440fea3f91f03f54c52de"
format_stamp: "Formatted at 2025-10-26 15:08:08 on dist-test-slave-8pgz"
server_key: "87bd07773fdfaa3e8792bf1b0d5c8462"
server_key_iv: "fde301559d6fb64fa2ca28bf59677a75"
server_key_version: "encryptionkey@0"
I20251026 15:08:08.198037 20989 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "61475c236ea440fea3f91f03f54c52de"
format_stamp: "Formatted at 2025-10-26 15:08:08 on dist-test-slave-8pgz"
server_key: "87bd07773fdfaa3e8792bf1b0d5c8462"
server_key_iv: "fde301559d6fb64fa2ca28bf59677a75"
server_key_version: "encryptionkey@0"
I20251026 15:08:08.199441 20989 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:08:08.200259 21003 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:08.200515 20989 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:08.200596 20989 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "61475c236ea440fea3f91f03f54c52de"
format_stamp: "Formatted at 2025-10-26 15:08:08 on dist-test-slave-8pgz"
server_key: "87bd07773fdfaa3e8792bf1b0d5c8462"
server_key_iv: "fde301559d6fb64fa2ca28bf59677a75"
server_key_version: "encryptionkey@0"
I20251026 15:08:08.200666 20989 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:08.208947 20989 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:08.209249 20989 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:08.209367 20989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:08.213438 20989 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:38201
I20251026 15:08:08.213469 21055 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:38201 every 8 connection(s)
I20251026 15:08:08.213789 20989 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:08.214442 21056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:08.216936 21056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de: Bootstrap starting.
I20251026 15:08:08.217474 21056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:08.217682 21056 log.cc:826] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:08.218495 21056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de: No bootstrap required, opened a new log
I20251026 15:08:08.219672 21056 raft_consensus.cc:359] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61475c236ea440fea3f91f03f54c52de" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38201 } }
I20251026 15:08:08.219834 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 20989
I20251026 15:08:08.219846 21056 raft_consensus.cc:385] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:08.219878 21056 raft_consensus.cc:740] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61475c236ea440fea3f91f03f54c52de, State: Initialized, Role: FOLLOWER
I20251026 15:08:08.219937 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:08:08.220064 21056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61475c236ea440fea3f91f03f54c52de" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38201 } }
I20251026 15:08:08.220105 15185 external_mini_cluster.cc:1468] Setting key ad972d5d15f58014adb895312776ae48
I20251026 15:08:08.220145 21056 raft_consensus.cc:399] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:08.220201 21056 raft_consensus.cc:493] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:08.220249 21056 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:08.220963 21056 raft_consensus.cc:515] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61475c236ea440fea3f91f03f54c52de" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38201 } }
I20251026 15:08:08.221078 21056 leader_election.cc:304] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61475c236ea440fea3f91f03f54c52de; no voters:
I20251026 15:08:08.221236 21056 leader_election.cc:290] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:08:08.221323 21061 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:08.221488 21061 raft_consensus.cc:697] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [term 1 LEADER]: Becoming Leader. State: Replica: 61475c236ea440fea3f91f03f54c52de, State: Running, Role: LEADER
I20251026 15:08:08.221495 21056 sys_catalog.cc:565] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:08.221601 21061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61475c236ea440fea3f91f03f54c52de" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38201 } }
I20251026 15:08:08.221954 21061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61475c236ea440fea3f91f03f54c52de. Latest consensus state: current_term: 1 leader_uuid: "61475c236ea440fea3f91f03f54c52de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61475c236ea440fea3f91f03f54c52de" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38201 } } }
I20251026 15:08:08.222036 21061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:08.221976 21063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61475c236ea440fea3f91f03f54c52de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61475c236ea440fea3f91f03f54c52de" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38201 } } }
I20251026 15:08:08.222188 21063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de [sys.catalog]: This master's current role is: LEADER
W20251026 15:08:08.223745 21076 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:08:08.223805 21076 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:08:08.223891 21066 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:08.224108 21066 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:08:08.225430 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:08.220556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.14.212.126:38201)
1026 15:08:08.220795 (+ 239us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:08.220802 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:08.220827 (+ 25us) server_negotiation.cc:408] Connection header received
1026 15:08:08.220943 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:08.220949 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:08.221011 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:08.221147 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:08.221732 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:08.223088 (+ 1356us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:08.224142 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:08.224369 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:08.224713 (+ 344us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:08.224721 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:08.224721 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:08.224956 (+ 235us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:08.224988 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:08.224997 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:08.225071 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:08.225212 (+ 141us) server_negotiation.cc:300] Negotiation successful
1026 15:08:08.225281 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20251026 15:08:08.225544 21066 catalog_manager.cc:1357] Generated new cluster ID: a9064f2f200145c0825f58aeaba1e0c0
I20251026 15:08:08.225742 21066 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:08:08.226588 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:08:08.233951 21066 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:08:08.235088 21066 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:08.238281 21066 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 61475c236ea440fea3f91f03f54c52de: Generated new TSK 0
I20251026 15:08:08.238489 21066 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:08:08.267020 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:08.247537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.14.212.126:38201)
1026 15:08:08.247610 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:08.247612 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:08.247623 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:08:08.247741 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:08.247742 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:08.247756 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:08.247828 (+ 72us) server_negotiation.cc:227] Negotiated authn=JWT
1026 15:08:08.248153 (+ 325us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:08.248978 (+ 825us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:08.249593 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:08.249795 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:08.249849 (+ 54us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1026 15:08:08.266684 (+ 16835us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
1026 15:08:08.266766 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:08.266886 (+ 120us) server_negotiation.cc:300] Negotiation successful
1026 15:08:08.266923 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47,"thread_start_us":117,"threads_started":1}
I20251026 15:08:08.274595 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:08.271799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36670 (local address 127.14.212.126:38201)
1026 15:08:08.271862 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:08.271865 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:08.271904 (+ 39us) server_negotiation.cc:408] Connection header received
1026 15:08:08.271990 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:08.271991 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:08.272000 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:08.272064 (+ 64us) server_negotiation.cc:227] Negotiated authn=JWT
1026 15:08:08.272285 (+ 221us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:08.273062 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:08.273690 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:08.273879 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:08.273903 (+ 24us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1026 15:08:08.274368 (+ 465us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1026 15:08:08.274529 (+ 161us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36670: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":37}
W20251026 15:08:08.274678 21060 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36670: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20251026 15:08:08.274843 21092 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:08.271706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:38201 (local address 127.0.0.1:36670)
1026 15:08:08.271843 (+ 137us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:08.271853 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:08.271925 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:08.272071 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:08.272072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:08.272074 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
1026 15:08:08.272227 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:08.272229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:08.273199 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:08.273200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:08.273606 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:08.273608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:08.273678 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:08.273682 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1026 15:08:08.274521 (+ 839us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1026 15:08:08.274615 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.212.126:38201: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20251026 15:08:23.280496 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:23.278208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.14.212.126:38201)
1026 15:08:23.278356 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:23.278358 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:23.278376 (+ 18us) server_negotiation.cc:408] Connection header received
1026 15:08:23.278432 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:23.278433 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:23.278445 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:23.278508 (+ 63us) server_negotiation.cc:227] Negotiated authn=JWT
1026 15:08:23.278790 (+ 282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.279356 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:23.279941 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.280109 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:23.280130 (+ 21us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1026 15:08:23.280305 (+ 175us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1026 15:08:23.280417 (+ 112us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58374: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
W20251026 15:08:23.280560 21097 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:23.278070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:38201 (local address 127.0.0.1:58374)
1026 15:08:23.278250 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:23.278261 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:23.278343 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:23.278521 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:23.278522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:23.278524 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
1026 15:08:23.278709 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:23.278711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.279443 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:23.279445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:23.279858 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:23.279860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.279971 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:23.279977 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1026 15:08:23.280382 (+ 405us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1026 15:08:23.280474 (+ 92us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.212.126:38201: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
W20251026 15:08:23.280571 21098 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58374: JWT verification failed: token expired
W20251026 15:08:23.282070 21098 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:23.281592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.14.212.126:38201)
1026 15:08:23.281641 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:23.281643 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:23.281691 (+ 48us) server_negotiation.cc:408] Connection header received
1026 15:08:23.281767 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:23.281768 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:23.281781 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:23.281831 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:23.282013 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58376: BlockingRecv error: recv got EOF from 127.0.0.1:58376 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20251026 15:08:23.284698 21108 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.14.212.126:38201
W20251026 15:08:23.285095 21098 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:23.284561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58388 (local address 127.14.212.126:38201)
1026 15:08:23.284609 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:23.284611 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:23.284676 (+ 65us) server_negotiation.cc:408] Connection header received
1026 15:08:23.284836 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:23.284837 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:23.284848 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:23.284895 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:23.285037 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58388: BlockingRecv error: recv got EOF from 127.0.0.1:58388 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20251026 15:08:23.285126 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 20989
2025-10-26T15:08:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20251026 15:08:23.323925 15185 mini_oidc.cc:150] Starting JWKS server
I20251026 15:08:23.324673 15185 webserver.cc:492] Webserver started at https://127.0.0.1:33669/ using document root <none> and password file <none>
I20251026 15:08:23.324717 15185 mini_oidc.cc:186] Starting OIDC Discovery server
I20251026 15:08:23.324807 15185 webserver.cc:492] Webserver started at http://127.0.0.1:34927/ using document root <none> and password file <none>
2025-10-26T15:08:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:23Z Disabled control of system clock
I20251026 15:08:23.331713 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:39589
--webserver_interface=127.14.212.126
--webserver_port=0
--builtin_ntp_servers=127.14.212.84:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:39589
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33669/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/testchainca.cert with env {}
W20251026 15:08:23.410033 21114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:23.410218 21114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:23.410252 21114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:23.411685 21114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:23.411729 21114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:23.411751 21114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:23.411767 21114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:23.411787 21114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251026 15:08:23.411813 21114 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251026 15:08:23.411844 21114 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33669/jwks.json
I20251026 15:08:23.413477 21114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:39589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:39589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33669/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:23.413729 21114 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:23.413894 21114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:23.416954 21119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:23.416961 21120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:23.416961 21122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:23.417034 21114 server_base.cc:1047] running on GCE node
I20251026 15:08:23.417536 21114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:23.417829 21114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:23.418979 21114 hybrid_clock.cc:648] HybridClock initialized: now 1761491303418948 us; error 47 us; skew 500 ppm
I20251026 15:08:23.420145 21114 webserver.cc:492] Webserver started at http://127.14.212.126:40937/ using document root <none> and password file <none>
I20251026 15:08:23.420368 21114 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:23.420415 21114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:23.420526 21114 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:23.421470 21114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "773d8dff7009436090eecbeb00f899e9"
format_stamp: "Formatted at 2025-10-26 15:08:23 on dist-test-slave-8pgz"
server_key: "97f145f32d08edab51be39304c54391f"
server_key_iv: "36adeaf8fb0f01802d06da12cd9d923e"
server_key_version: "encryptionkey@0"
I20251026 15:08:23.421943 21114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "773d8dff7009436090eecbeb00f899e9"
format_stamp: "Formatted at 2025-10-26 15:08:23 on dist-test-slave-8pgz"
server_key: "97f145f32d08edab51be39304c54391f"
server_key_iv: "36adeaf8fb0f01802d06da12cd9d923e"
server_key_version: "encryptionkey@0"
I20251026 15:08:23.423249 21114 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:08:23.424101 21128 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:23.424293 21114 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:23.424372 21114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "773d8dff7009436090eecbeb00f899e9"
format_stamp: "Formatted at 2025-10-26 15:08:23 on dist-test-slave-8pgz"
server_key: "97f145f32d08edab51be39304c54391f"
server_key_iv: "36adeaf8fb0f01802d06da12cd9d923e"
server_key_version: "encryptionkey@0"
I20251026 15:08:23.424438 21114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:23.436280 21114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:23.436551 21114 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:23.436642 21114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:23.440228 21114 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:39589
I20251026 15:08:23.440291 21180 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:39589 every 8 connection(s)
I20251026 15:08:23.440572 21114 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:23.441149 21181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:23.443506 21181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9: Bootstrap starting.
I20251026 15:08:23.444059 21181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:23.444296 21181 log.cc:826] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:23.444969 21181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9: No bootstrap required, opened a new log
I20251026 15:08:23.445369 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 21114
I20251026 15:08:23.445451 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:08:23.445604 15185 external_mini_cluster.cc:1468] Setting key bddb6fd90722c7817b94131a667e1335
I20251026 15:08:23.446461 21181 raft_consensus.cc:359] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773d8dff7009436090eecbeb00f899e9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39589 } }
I20251026 15:08:23.446584 21181 raft_consensus.cc:385] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:23.446616 21181 raft_consensus.cc:740] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773d8dff7009436090eecbeb00f899e9, State: Initialized, Role: FOLLOWER
I20251026 15:08:23.446713 21181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773d8dff7009436090eecbeb00f899e9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39589 } }
I20251026 15:08:23.446787 21181 raft_consensus.cc:399] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:23.446820 21181 raft_consensus.cc:493] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:23.446863 21181 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:23.447502 21181 raft_consensus.cc:515] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773d8dff7009436090eecbeb00f899e9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39589 } }
I20251026 15:08:23.447592 21181 leader_election.cc:304] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 773d8dff7009436090eecbeb00f899e9; no voters:
I20251026 15:08:23.447722 21181 leader_election.cc:290] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:08:23.447772 21186 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:23.447906 21186 raft_consensus.cc:697] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [term 1 LEADER]: Becoming Leader. State: Replica: 773d8dff7009436090eecbeb00f899e9, State: Running, Role: LEADER
I20251026 15:08:23.447937 21181 sys_catalog.cc:565] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:23.448020 21186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773d8dff7009436090eecbeb00f899e9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39589 } }
I20251026 15:08:23.448308 21188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "773d8dff7009436090eecbeb00f899e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773d8dff7009436090eecbeb00f899e9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39589 } } }
I20251026 15:08:23.448482 21188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:23.448740 21194 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:23.448293 21186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 773d8dff7009436090eecbeb00f899e9. Latest consensus state: current_term: 1 leader_uuid: "773d8dff7009436090eecbeb00f899e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773d8dff7009436090eecbeb00f899e9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39589 } } }
I20251026 15:08:23.448941 21186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:23.449774 21194 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:08:23.450551 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:23.446261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.14.212.126:39589)
1026 15:08:23.446470 (+ 209us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:23.446477 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:23.446495 (+ 18us) server_negotiation.cc:408] Connection header received
1026 15:08:23.446610 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:23.446615 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:23.446674 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:23.446800 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:23.447287 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.448305 (+ 1018us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:23.449202 (+ 897us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.449475 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:23.449878 (+ 403us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:23.449886 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:23.449886 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:23.450131 (+ 245us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:23.450163 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:23.450171 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:23.450249 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:23.450326 (+ 77us) server_negotiation.cc:300] Negotiation successful
1026 15:08:23.450383 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20251026 15:08:23.451534 21194 catalog_manager.cc:1357] Generated new cluster ID: 6a40de3fe60a42c5ad4a53853a0555de
I20251026 15:08:23.451558 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:08:23.451592 21194 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:08:23.455842 21194 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:08:23.456348 21194 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:23.460007 21194 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 773d8dff7009436090eecbeb00f899e9: Generated new TSK 0
I20251026 15:08:23.460165 21194 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251026 15:08:23.486146 21211 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251026 15:08:23.486229 21211 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251026 15:08:23.486567 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:23.471853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.14.212.126:39589)
1026 15:08:23.471907 (+ 54us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:23.471909 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:23.471972 (+ 63us) server_negotiation.cc:408] Connection header received
1026 15:08:23.472036 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:23.472037 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:23.472050 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:23.472123 (+ 73us) server_negotiation.cc:227] Negotiated authn=JWT
1026 15:08:23.472342 (+ 219us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.473127 (+ 785us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:23.473713 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.473893 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:23.473947 (+ 54us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1026 15:08:23.486317 (+ 12370us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1026 15:08:23.486495 (+ 178us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":34}
W20251026 15:08:23.486635 21185 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251026 15:08:23.486636 21210 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:23.471771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:39589 (local address 127.0.0.1:50230)
1026 15:08:23.471903 (+ 132us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:23.471914 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:23.471977 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:23.472124 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:23.472125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:23.472127 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
1026 15:08:23.472286 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:23.472288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.473224 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:23.473226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:23.473640 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:23.473642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.473711 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:23.473716 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1026 15:08:23.486446 (+ 12730us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1026 15:08:23.486552 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.212.126:39589: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20251026 15:08:23.487012 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 21114
2025-10-26T15:08:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (200 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20251026 15:08:23.524621 15185 mini_oidc.cc:150] Starting JWKS server
I20251026 15:08:23.525358 15185 webserver.cc:492] Webserver started at https://127.0.0.1:40203/ using document root <none> and password file <none>
I20251026 15:08:23.525403 15185 mini_oidc.cc:186] Starting OIDC Discovery server
I20251026 15:08:23.525511 15185 webserver.cc:492] Webserver started at http://127.0.0.1:35527/ using document root <none> and password file <none>
2025-10-26T15:08:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:23Z Disabled control of system clock
I20251026 15:08:23.532589 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:42893
--webserver_interface=127.14.212.126
--webserver_port=0
--builtin_ntp_servers=127.14.212.84:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:42893
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40203/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251026 15:08:23.613116 21217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:23.613291 21217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:23.613313 21217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:23.614807 21217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:23.614857 21217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:23.614871 21217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:23.614883 21217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:23.614895 21217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251026 15:08:23.614907 21217 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251026 15:08:23.614918 21217 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40203/jwks.json
I20251026 15:08:23.616490 21217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:42893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:42893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40203/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:23.616704 21217 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:23.616900 21217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:23.619979 21222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:23.619979 21225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:23.619983 21223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:23.620256 21217 server_base.cc:1047] running on GCE node
I20251026 15:08:23.620476 21217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:23.620702 21217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:23.621836 21217 hybrid_clock.cc:648] HybridClock initialized: now 1761491303621825 us; error 38 us; skew 500 ppm
I20251026 15:08:23.623111 21217 webserver.cc:492] Webserver started at http://127.14.212.126:38679/ using document root <none> and password file <none>
I20251026 15:08:23.623332 21217 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:23.623375 21217 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:23.623482 21217 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:23.624475 21217 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "036482fccde745cea8eacb2c09288b90"
format_stamp: "Formatted at 2025-10-26 15:08:23 on dist-test-slave-8pgz"
server_key: "c055ec669de75b5d359175e06a75fb62"
server_key_iv: "4b17fe936935fbf6bc8c1ee44f10a16e"
server_key_version: "encryptionkey@0"
I20251026 15:08:23.624775 21217 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "036482fccde745cea8eacb2c09288b90"
format_stamp: "Formatted at 2025-10-26 15:08:23 on dist-test-slave-8pgz"
server_key: "c055ec669de75b5d359175e06a75fb62"
server_key_iv: "4b17fe936935fbf6bc8c1ee44f10a16e"
server_key_version: "encryptionkey@0"
I20251026 15:08:23.626035 21217 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:08:23.626890 21231 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:23.627069 21217 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251026 15:08:23.627152 21217 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "036482fccde745cea8eacb2c09288b90"
format_stamp: "Formatted at 2025-10-26 15:08:23 on dist-test-slave-8pgz"
server_key: "c055ec669de75b5d359175e06a75fb62"
server_key_iv: "4b17fe936935fbf6bc8c1ee44f10a16e"
server_key_version: "encryptionkey@0"
I20251026 15:08:23.627218 21217 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:23.640220 21217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:23.640513 21217 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:23.640635 21217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:23.644446 21217 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:42893
I20251026 15:08:23.644483 21283 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:42893 every 8 connection(s)
I20251026 15:08:23.644795 21217 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:23.645416 21284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:23.646178 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 21217
I20251026 15:08:23.646284 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:08:23.646458 15185 external_mini_cluster.cc:1468] Setting key ea7fc64cb7cd71771fbb5fca405fd148
I20251026 15:08:23.648164 21284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90: Bootstrap starting.
I20251026 15:08:23.648691 21284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:23.648937 21284 log.cc:826] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:23.649605 21284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90: No bootstrap required, opened a new log
I20251026 15:08:23.650839 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:23.646993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.14.212.126:42893)
1026 15:08:23.647211 (+ 218us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:23.647217 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:23.647233 (+ 16us) server_negotiation.cc:408] Connection header received
1026 15:08:23.647538 (+ 305us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:23.647543 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:23.647596 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:23.647718 (+ 122us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:23.648095 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.648856 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:23.649566 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.649811 (+ 245us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:23.650261 (+ 450us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:23.650269 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:23.650270 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:23.650463 (+ 193us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:23.650489 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:23.650496 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:23.650563 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:23.650645 (+ 82us) server_negotiation.cc:300] Negotiation successful
1026 15:08:23.650700 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20251026 15:08:23.651039 21284 raft_consensus.cc:359] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036482fccde745cea8eacb2c09288b90" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42893 } }
I20251026 15:08:23.651183 21284 raft_consensus.cc:385] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:23.651225 21284 raft_consensus.cc:740] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036482fccde745cea8eacb2c09288b90, State: Initialized, Role: FOLLOWER
I20251026 15:08:23.651309 21284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036482fccde745cea8eacb2c09288b90" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42893 } }
I20251026 15:08:23.651376 21284 raft_consensus.cc:399] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:23.651412 21284 raft_consensus.cc:493] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:23.651459 21284 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:23.651787 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:08:23.652115 21284 raft_consensus.cc:515] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036482fccde745cea8eacb2c09288b90" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42893 } }
I20251026 15:08:23.652210 21284 leader_election.cc:304] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 036482fccde745cea8eacb2c09288b90; no voters:
I20251026 15:08:23.652360 21284 leader_election.cc:290] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:08:23.652410 21289 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:23.652516 21289 raft_consensus.cc:697] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [term 1 LEADER]: Becoming Leader. State: Replica: 036482fccde745cea8eacb2c09288b90, State: Running, Role: LEADER
I20251026 15:08:23.652592 21284 sys_catalog.cc:565] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:23.652603 21289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036482fccde745cea8eacb2c09288b90" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42893 } }
I20251026 15:08:23.652933 21289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 036482fccde745cea8eacb2c09288b90. Latest consensus state: current_term: 1 leader_uuid: "036482fccde745cea8eacb2c09288b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036482fccde745cea8eacb2c09288b90" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42893 } } }
I20251026 15:08:23.653008 21289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:23.652943 21291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "036482fccde745cea8eacb2c09288b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036482fccde745cea8eacb2c09288b90" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42893 } } }
I20251026 15:08:23.653189 21291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:23.653240 21294 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:23.654008 21294 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:08:23.655520 21294 catalog_manager.cc:1357] Generated new cluster ID: cb06db9c7eed40e488772057ed9f0427
I20251026 15:08:23.655582 21294 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:08:23.665853 21294 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:08:23.666469 21294 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:23.670701 21294 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 036482fccde745cea8eacb2c09288b90: Generated new TSK 0
I20251026 15:08:23.670859 21294 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251026 15:08:23.699693 21314 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251026 15:08:23.699771 21314 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251026 15:08:23.700897 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:23.683218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.14.212.126:42893)
1026 15:08:23.683264 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:23.683266 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:23.683344 (+ 78us) server_negotiation.cc:408] Connection header received
1026 15:08:23.683395 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:23.683396 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:23.683408 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:23.683479 (+ 71us) server_negotiation.cc:227] Negotiated authn=JWT
1026 15:08:23.683749 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.684570 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:23.685189 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.685383 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:23.685441 (+ 58us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1026 15:08:23.700608 (+ 15167us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40203/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1026 15:08:23.700814 (+ 206us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37188: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40203/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20251026 15:08:23.700953 21287 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37188: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40203/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251026 15:08:23.701807 21313 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:23.683098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:42893 (local address 127.0.0.1:37188)
1026 15:08:23.683273 (+ 175us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:23.683282 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:23.683340 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:23.683513 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:23.683514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:23.683516 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
1026 15:08:23.683683 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:23.683684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.684673 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:23.684674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:23.685116 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:23.685118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.685184 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:23.685188 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1026 15:08:23.700736 (+ 15548us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40203/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1026 15:08:23.701675 (+ 939us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.212.126:42893: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40203/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
I20251026 15:08:23.702100 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 21217
2025-10-26T15:08:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-10-26T15:08:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:23Z Disabled control of system clock
I20251026 15:08:23.748338 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46731
--webserver_interface=127.14.212.126
--webserver_port=0
--builtin_ntp_servers=127.14.212.84:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:46731
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/testchainca.cert with env {}
W20251026 15:08:23.825616 21318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:23.825811 21318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:23.825848 21318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:23.827378 21318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:23.827431 21318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:23.827450 21318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:23.827474 21318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:23.827491 21318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251026 15:08:23.827513 21318 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251026 15:08:23.827533 21318 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20251026 15:08:23.829265 21318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:46731
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:23.829521 21318 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:23.829696 21318 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:23.832338 21324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:23.832345 21323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:23.832520 21318 server_base.cc:1047] running on GCE node
W20251026 15:08:23.832345 21326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:23.832811 21318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:23.833048 21318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:23.834179 21318 hybrid_clock.cc:648] HybridClock initialized: now 1761491303834162 us; error 36 us; skew 500 ppm
I20251026 15:08:23.835379 21318 webserver.cc:492] Webserver started at http://127.14.212.126:45651/ using document root <none> and password file <none>
I20251026 15:08:23.835577 21318 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:23.835624 21318 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:23.835737 21318 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:23.836685 21318 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "43ca52b8991f49a5a1db3087944a3069"
format_stamp: "Formatted at 2025-10-26 15:08:23 on dist-test-slave-8pgz"
server_key: "a753431466bce90e45f905dca44663a2"
server_key_iv: "cfe52cdb7533f56e440983756b821afa"
server_key_version: "encryptionkey@0"
I20251026 15:08:23.836995 21318 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "43ca52b8991f49a5a1db3087944a3069"
format_stamp: "Formatted at 2025-10-26 15:08:23 on dist-test-slave-8pgz"
server_key: "a753431466bce90e45f905dca44663a2"
server_key_iv: "cfe52cdb7533f56e440983756b821afa"
server_key_version: "encryptionkey@0"
I20251026 15:08:23.838227 21318 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:08:23.839071 21332 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:23.839334 21318 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:23.839411 21318 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "43ca52b8991f49a5a1db3087944a3069"
format_stamp: "Formatted at 2025-10-26 15:08:23 on dist-test-slave-8pgz"
server_key: "a753431466bce90e45f905dca44663a2"
server_key_iv: "cfe52cdb7533f56e440983756b821afa"
server_key_version: "encryptionkey@0"
I20251026 15:08:23.839474 21318 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:23.850907 21318 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:23.851231 21318 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:23.851368 21318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:23.855156 21318 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:46731
I20251026 15:08:23.855233 21384 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:46731 every 8 connection(s)
I20251026 15:08:23.855545 21318 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:23.856156 21385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:23.858739 21385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069: Bootstrap starting.
I20251026 15:08:23.859344 21385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:23.859573 21385 log.cc:826] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:23.860270 21385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069: No bootstrap required, opened a new log
I20251026 15:08:23.861641 21385 raft_consensus.cc:359] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ca52b8991f49a5a1db3087944a3069" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46731 } }
I20251026 15:08:23.861784 21385 raft_consensus.cc:385] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:23.861822 21385 raft_consensus.cc:740] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ca52b8991f49a5a1db3087944a3069, State: Initialized, Role: FOLLOWER
I20251026 15:08:23.861964 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 21318
I20251026 15:08:23.861974 21385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ca52b8991f49a5a1db3087944a3069" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46731 } }
I20251026 15:08:23.862054 21385 raft_consensus.cc:399] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:23.862063 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:08:23.862099 21385 raft_consensus.cc:493] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:23.862151 21385 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:23.862221 15185 external_mini_cluster.cc:1468] Setting key 8d79693e4c96c3246fd32ff68e6c4988
I20251026 15:08:23.862893 21385 raft_consensus.cc:515] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ca52b8991f49a5a1db3087944a3069" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46731 } }
I20251026 15:08:23.863051 21385 leader_election.cc:304] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43ca52b8991f49a5a1db3087944a3069; no voters:
I20251026 15:08:23.863200 21385 leader_election.cc:290] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:08:23.863315 21390 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:23.863445 21385 sys_catalog.cc:565] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:23.863479 21390 raft_consensus.cc:697] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [term 1 LEADER]: Becoming Leader. State: Replica: 43ca52b8991f49a5a1db3087944a3069, State: Running, Role: LEADER
I20251026 15:08:23.863644 21390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ca52b8991f49a5a1db3087944a3069" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46731 } }
I20251026 15:08:23.864811 21392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43ca52b8991f49a5a1db3087944a3069. Latest consensus state: current_term: 1 leader_uuid: "43ca52b8991f49a5a1db3087944a3069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ca52b8991f49a5a1db3087944a3069" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46731 } } }
I20251026 15:08:23.864903 21392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:23.865093 21391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43ca52b8991f49a5a1db3087944a3069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ca52b8991f49a5a1db3087944a3069" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46731 } } }
I20251026 15:08:23.866536 21391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069 [sys.catalog]: This master's current role is: LEADER
W20251026 15:08:23.865381 21405 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:08:23.866608 21405 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:08:23.866663 21406 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:23.866847 21406 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:08:23.867273 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:23.862740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.14.212.126:46731)
1026 15:08:23.862926 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:23.862933 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:23.862957 (+ 24us) server_negotiation.cc:408] Connection header received
1026 15:08:23.863071 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:23.863075 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:23.863132 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:23.863281 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:23.863714 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.864871 (+ 1157us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:23.865661 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.865915 (+ 254us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:23.866420 (+ 505us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:23.866427 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:23.866428 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:23.866675 (+ 247us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:23.866708 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:23.866716 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:23.866789 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:23.866916 (+ 127us) server_negotiation.cc:300] Negotiation successful
1026 15:08:23.867122 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20251026 15:08:23.868199 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:08:23.868826 21406 catalog_manager.cc:1357] Generated new cluster ID: 1379052289314bff80147b7a96f48270
I20251026 15:08:23.868884 21406 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:08:23.879297 21406 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:08:23.879853 21406 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:23.889536 21406 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 43ca52b8991f49a5a1db3087944a3069: Generated new TSK 0
I20251026 15:08:23.889710 21406 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:08:23.943622 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:23.895317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35876 (local address 127.14.212.126:46731)
1026 15:08:23.895371 (+ 54us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:23.895373 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:23.895486 (+ 113us) server_negotiation.cc:408] Connection header received
1026 15:08:23.895567 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:23.895567 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:23.895578 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:23.895627 (+ 49us) server_negotiation.cc:227] Negotiated authn=JWT
1026 15:08:23.895975 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.896463 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:23.897294 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.897419 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:23.897456 (+ 37us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1026 15:08:23.943331 (+ 45875us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1026 15:08:23.943516 (+ 185us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35876: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":30}
W20251026 15:08:23.943689 21389 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35876: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20251026 15:08:23.943730 21414 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:23.895251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46731 (local address 127.0.0.1:35876)
1026 15:08:23.895411 (+ 160us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:23.895423 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:23.895497 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:23.895648 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:23.895649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:23.895651 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
1026 15:08:23.895890 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:23.895892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.896545 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:23.896546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:23.897209 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:23.897211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:23.897295 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:23.897299 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1026 15:08:23.943464 (+ 46165us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1026 15:08:23.943610 (+ 146us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.212.126:46731: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20251026 15:08:23.944316 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 21318
2025-10-26T15:08:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (220 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:08:24 dist-test-slave-8pgz krb5kdc[21426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:08:24 dist-test-slave-8pgz krb5kdc[21426](info): set up 2 sockets
Oct 26 15:08:24 dist-test-slave-8pgz krb5kdc[21426](info): commencing operation
krb5kdc: starting...
W20251026 15:08:26.112149 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.974s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:08:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:26.245442 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:44653
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:44653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:26.327361 21442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:26.327572 21442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:26.327613 21442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:26.329084 21442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:26.329138 21442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:26.329166 21442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:26.329191 21442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:26.329212 21442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:08:26.330893 21442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:44653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:44653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:26.331182 21442 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:26.331382 21442 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:26.333971 21447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:26.333953 21450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:26.334092 21448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:26.334204 21442 server_base.cc:1047] running on GCE node
I20251026 15:08:26.334518 21442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:26.334734 21442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:26.335867 21442 hybrid_clock.cc:648] HybridClock initialized: now 1761491306335848 us; error 40 us; skew 500 ppm
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:26.338246 21442 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:08:26.338802 21442 webserver.cc:492] Webserver started at http://127.14.212.126:37083/ using document root <none> and password file <none>
I20251026 15:08:26.339002 21442 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:26.339043 21442 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:26.339155 21442 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:26.340178 21442 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "f6d44868677f4142a5b9689733f49b44"
format_stamp: "Formatted at 2025-10-26 15:08:26 on dist-test-slave-8pgz"
server_key: "b1ddd1c842d2445662aaa339d2e0ca14"
server_key_iv: "4814ef6c1273c801849aecf1fd4dec07"
server_key_version: "encryptionkey@0"
I20251026 15:08:26.340474 21442 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "f6d44868677f4142a5b9689733f49b44"
format_stamp: "Formatted at 2025-10-26 15:08:26 on dist-test-slave-8pgz"
server_key: "b1ddd1c842d2445662aaa339d2e0ca14"
server_key_iv: "4814ef6c1273c801849aecf1fd4dec07"
server_key_version: "encryptionkey@0"
I20251026 15:08:26.341722 21442 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:08:26.342518 21457 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:26.342685 21442 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:26.342762 21442 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "f6d44868677f4142a5b9689733f49b44"
format_stamp: "Formatted at 2025-10-26 15:08:26 on dist-test-slave-8pgz"
server_key: "b1ddd1c842d2445662aaa339d2e0ca14"
server_key_iv: "4814ef6c1273c801849aecf1fd4dec07"
server_key_version: "encryptionkey@0"
I20251026 15:08:26.342825 21442 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:26.349383 21442 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:26.351617 21442 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:26.351760 21442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:26.355803 21442 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:44653
I20251026 15:08:26.355852 21509 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:44653 every 8 connection(s)
I20251026 15:08:26.356146 21442 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:26.356739 21510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:26.359290 21510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44: Bootstrap starting.
I20251026 15:08:26.359819 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 21442
I20251026 15:08:26.359850 21510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:26.359910 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:08:26.360075 15185 external_mini_cluster.cc:1468] Setting key 9bf7fbe268f86e7c48808913f8cae03e
I20251026 15:08:26.360090 21510 log.cc:826] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:26.360800 21510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44: No bootstrap required, opened a new log
I20251026 15:08:26.362087 21510 raft_consensus.cc:359] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d44868677f4142a5b9689733f49b44" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44653 } }
I20251026 15:08:26.362214 21510 raft_consensus.cc:385] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:26.362250 21510 raft_consensus.cc:740] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d44868677f4142a5b9689733f49b44, State: Initialized, Role: FOLLOWER
I20251026 15:08:26.362355 21510 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d44868677f4142a5b9689733f49b44" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44653 } }
I20251026 15:08:26.362455 21510 raft_consensus.cc:399] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:26.362500 21510 raft_consensus.cc:493] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:26.362556 21510 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:26.363147 21510 raft_consensus.cc:515] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d44868677f4142a5b9689733f49b44" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44653 } }
I20251026 15:08:26.363265 21510 leader_election.cc:304] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d44868677f4142a5b9689733f49b44; no voters:
I20251026 15:08:26.363409 21510 leader_election.cc:290] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:08:26.363477 21515 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:26.363632 21510 sys_catalog.cc:565] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:26.363682 21515 raft_consensus.cc:697] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [term 1 LEADER]: Becoming Leader. State: Replica: f6d44868677f4142a5b9689733f49b44, State: Running, Role: LEADER
I20251026 15:08:26.363811 21515 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d44868677f4142a5b9689733f49b44" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44653 } }
I20251026 15:08:26.364277 21516 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6d44868677f4142a5b9689733f49b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d44868677f4142a5b9689733f49b44" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44653 } } }
I20251026 15:08:26.364775 21516 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:26.365032 21527 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:26.365131 21517 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6d44868677f4142a5b9689733f49b44. Latest consensus state: current_term: 1 leader_uuid: "f6d44868677f4142a5b9689733f49b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d44868677f4142a5b9689733f49b44" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44653 } } }
I20251026 15:08:26.365206 21517 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:26.365585 21527 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:26.366947 21527 catalog_manager.cc:1357] Generated new cluster ID: 2df61079c4f5456ab2e7827444c1110e
I20251026 15:08:26.366989 21527 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:08:26.370743 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:26.360596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59150 (local address 127.14.212.126:44653)
1026 15:08:26.360777 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:26.360786 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:26.360804 (+ 18us) server_negotiation.cc:408] Connection header received
1026 15:08:26.361159 (+ 355us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:26.361164 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:26.361210 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:26.361337 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:26.362099 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.362964 (+ 865us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:26.363836 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.364091 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:26.367196 (+ 3105us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:26.367201 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:26.367202 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:26.367222 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:26.369313 (+ 2091us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.369704 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.369707 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.369707 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.369758 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.369904 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.369904 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.369905 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.370134 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:26.370274 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:26.370426 (+ 152us) server_negotiation.cc:300] Negotiation successful
1026 15:08:26.370546 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20251026 15:08:26.372982 21527 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:08:26.373503 21527 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:26.376495 21527 catalog_manager.cc:6022] T 00000000000000000000000000000000 P f6d44868677f4142a5b9689733f49b44: Generated new TSK 0
I20251026 15:08:26.376675 21527 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:26.428395 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:44653
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:26.510567 21538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:26.510735 21538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:26.510758 21538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:26.512194 21538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:26.512243 21538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:26.512269 21538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:08:26.513859 21538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:44653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:26.514106 21538 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:26.514313 21538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:26.516983 21544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:26.517098 21538 server_base.cc:1047] running on GCE node
W20251026 15:08:26.517148 21543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:26.516968 21546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:26.517395 21538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:26.517647 21538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:26.518800 21538 hybrid_clock.cc:648] HybridClock initialized: now 1761491306518786 us; error 27 us; skew 500 ppm
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:26.521057 21538 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:08:26.521591 21538 webserver.cc:492] Webserver started at http://127.14.212.65:40739/ using document root <none> and password file <none>
I20251026 15:08:26.521781 21538 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:26.521822 21538 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:26.521936 21538 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:26.522913 21538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "6b69d33a697b44e3a4617fe85c39115a"
format_stamp: "Formatted at 2025-10-26 15:08:26 on dist-test-slave-8pgz"
server_key: "b44abf42a5c2e9e0390c08389340e4e2"
server_key_iv: "1feece633bc0fb9b1af8cfb4b81ee111"
server_key_version: "encryptionkey@0"
I20251026 15:08:26.523221 21538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "6b69d33a697b44e3a4617fe85c39115a"
format_stamp: "Formatted at 2025-10-26 15:08:26 on dist-test-slave-8pgz"
server_key: "b44abf42a5c2e9e0390c08389340e4e2"
server_key_iv: "1feece633bc0fb9b1af8cfb4b81ee111"
server_key_version: "encryptionkey@0"
I20251026 15:08:26.524397 21538 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:08:26.525064 21553 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:26.525233 21538 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:26.525311 21538 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "6b69d33a697b44e3a4617fe85c39115a"
format_stamp: "Formatted at 2025-10-26 15:08:26 on dist-test-slave-8pgz"
server_key: "b44abf42a5c2e9e0390c08389340e4e2"
server_key_iv: "1feece633bc0fb9b1af8cfb4b81ee111"
server_key_version: "encryptionkey@0"
I20251026 15:08:26.525377 21538 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:26.537355 21538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:26.539567 21538 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:26.539716 21538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:26.539930 21538 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:26.540244 21538 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:26.540277 21538 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:26.540313 21538 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:26.540342 21538 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:26.546128 21538 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:38011
I20251026 15:08:26.546166 21666 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:38011 every 8 connection(s)
I20251026 15:08:26.546515 21538 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:26.552943 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 21538
I20251026 15:08:26.553047 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:08:26.553206 15185 external_mini_cluster.cc:1468] Setting key 9e6095688fe8c3ca13262212b96acec8
I20251026 15:08:26.555451 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:26.547343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:37931 (local address 127.14.212.126:44653)
1026 15:08:26.547428 (+ 85us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:26.547433 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:26.547486 (+ 53us) server_negotiation.cc:408] Connection header received
1026 15:08:26.547968 (+ 482us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:26.547969 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:26.547979 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:26.548032 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:26.548749 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.549265 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:26.550231 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.550346 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:26.552346 (+ 2000us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:26.552350 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:26.552351 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:26.552357 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:26.554024 (+ 1667us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.554629 (+ 605us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.554630 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.554631 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.554658 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.554837 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.554838 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.554838 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.554932 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:26.554996 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:26.555258 (+ 262us) server_negotiation.cc:300] Negotiation successful
1026 15:08:26.555353 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251026 15:08:26.555549 21669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:26.547228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:44653 (local address 127.14.212.65:37931)
1026 15:08:26.547374 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:26.547409 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:26.547887 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:26.548079 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:26.548083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:26.548389 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:26.548676 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:26.548679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.549342 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:26.549343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:26.550168 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:26.550170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.550275 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:26.550823 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:26.550839 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:26.552238 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:26.554110 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:26.554116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:26.554119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:26.554549 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:26.554715 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:26.554716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:26.554717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:26.554781 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:26.554995 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:26.555001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:26.555127 (+ 126us) client_negotiation.cc:769] Sending connection context
1026 15:08:26.555226 (+ 99us) client_negotiation.cc:240] Negotiation successful
1026 15:08:26.555363 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20251026 15:08:26.556111 21667 heartbeater.cc:344] Connected to a master server at 127.14.212.126:44653
I20251026 15:08:26.556238 21667 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:26.556421 21667 heartbeater.cc:507] Master 127.14.212.126:44653 requested a full tablet report, sending...
I20251026 15:08:26.556924 21474 ts_manager.cc:194] Registered new tserver with Master: 6b69d33a697b44e3a4617fe85c39115a (127.14.212.65:38011)
I20251026 15:08:26.557561 21474 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:37931
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:26.597494 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:44653
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:26.677440 21674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:26.677624 21674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:26.677650 21674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:26.679165 21674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:26.679216 21674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:26.679243 21674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:08:26.680764 21674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:44653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:26.680987 21674 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:26.681172 21674 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:08:26.684269 21674 server_base.cc:1047] running on GCE node
W20251026 15:08:26.684211 21679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:26.684198 21680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:26.684206 21682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:26.684530 21674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:26.684708 21674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:26.685870 21674 hybrid_clock.cc:648] HybridClock initialized: now 1761491306685856 us; error 28 us; skew 500 ppm
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:26.688407 21674 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:08:26.688908 21674 webserver.cc:492] Webserver started at http://127.14.212.66:34109/ using document root <none> and password file <none>
I20251026 15:08:26.689128 21674 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:26.689167 21674 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:26.689272 21674 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:26.690215 21674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "c3c5cda4e90743f0a8eec57f48ffd315"
format_stamp: "Formatted at 2025-10-26 15:08:26 on dist-test-slave-8pgz"
server_key: "16ad3e9939a270a062bf6dddb68f74b1"
server_key_iv: "bfc77f6122251b2a33ee13de1b137534"
server_key_version: "encryptionkey@0"
I20251026 15:08:26.690536 21674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "c3c5cda4e90743f0a8eec57f48ffd315"
format_stamp: "Formatted at 2025-10-26 15:08:26 on dist-test-slave-8pgz"
server_key: "16ad3e9939a270a062bf6dddb68f74b1"
server_key_iv: "bfc77f6122251b2a33ee13de1b137534"
server_key_version: "encryptionkey@0"
I20251026 15:08:26.691771 21674 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:26.692526 21689 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:26.692718 21674 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:08:26.692786 21674 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "c3c5cda4e90743f0a8eec57f48ffd315"
format_stamp: "Formatted at 2025-10-26 15:08:26 on dist-test-slave-8pgz"
server_key: "16ad3e9939a270a062bf6dddb68f74b1"
server_key_iv: "bfc77f6122251b2a33ee13de1b137534"
server_key_version: "encryptionkey@0"
I20251026 15:08:26.692848 21674 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:26.710795 21674 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:26.712884 21674 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:26.713136 21674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:26.713392 21674 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:26.713730 21674 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:26.713764 21674 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:26.713799 21674 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:26.713829 21674 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:26.719893 21674 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:45433
I20251026 15:08:26.719941 21802 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:45433 every 8 connection(s)
I20251026 15:08:26.720268 21674 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:08:26.722064 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 21674
I20251026 15:08:26.722152 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:08:26.722332 15185 external_mini_cluster.cc:1468] Setting key 3c8714b313885a8a489547f79ca55e9b
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:26.730326 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:26.720927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:52067 (local address 127.14.212.126:44653)
1026 15:08:26.720979 (+ 52us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:26.720981 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:26.721142 (+ 161us) server_negotiation.cc:408] Connection header received
1026 15:08:26.721674 (+ 532us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:26.721676 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:26.721688 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:26.721763 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:26.722661 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.723377 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:26.724486 (+ 1109us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.724626 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:26.727133 (+ 2507us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:26.727140 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:26.727141 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:26.727150 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:26.729059 (+ 1909us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.729457 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.729458 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.729459 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.729485 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.729655 (+ 170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.729656 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.729656 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.729794 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:26.729873 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:26.730128 (+ 255us) server_negotiation.cc:300] Negotiation successful
1026 15:08:26.730231 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251026 15:08:26.730346 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:26.720880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:44653 (local address 127.14.212.66:52067)
1026 15:08:26.721031 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:26.721062 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:26.721557 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:26.721829 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:26.721833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:26.722220 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:26.722576 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:26.722580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.723468 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:26.723469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:26.724384 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:26.724387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.724558 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:26.725203 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:26.725234 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:26.727033 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:26.729146 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:26.729153 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:26.729156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:26.729379 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:26.729547 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:26.729548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:26.729548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:26.729605 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:26.729856 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:26.729859 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:26.729985 (+ 126us) client_negotiation.cc:769] Sending connection context
1026 15:08:26.730079 (+ 94us) client_negotiation.cc:240] Negotiation successful
1026 15:08:26.730194 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
I20251026 15:08:26.730957 21803 heartbeater.cc:344] Connected to a master server at 127.14.212.126:44653
I20251026 15:08:26.731074 21803 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:26.731333 21803 heartbeater.cc:507] Master 127.14.212.126:44653 requested a full tablet report, sending...
I20251026 15:08:26.731664 21474 ts_manager.cc:194] Registered new tserver with Master: c3c5cda4e90743f0a8eec57f48ffd315 (127.14.212.66:45433)
I20251026 15:08:26.732110 21474 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:52067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:26.769474 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:44653
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:26.851524 21810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:26.851707 21810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:26.851732 21810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:26.853147 21810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:26.853195 21810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:26.853222 21810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:08:26.854828 21810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:44653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:26.855036 21810 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:26.855191 21810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:26.858021 21816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:26.858000 21815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:26.857980 21818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:26.858064 21810 server_base.cc:1047] running on GCE node
I20251026 15:08:26.858446 21810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:26.858661 21810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:26.859818 21810 hybrid_clock.cc:648] HybridClock initialized: now 1761491306859790 us; error 47 us; skew 500 ppm
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:26.862334 21810 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:08:26.862924 21810 webserver.cc:492] Webserver started at http://127.14.212.67:40091/ using document root <none> and password file <none>
I20251026 15:08:26.863124 21810 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:26.863164 21810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:26.863279 21810 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:26.864346 21810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "03a2958b23f240b5b482844792cc6ed1"
format_stamp: "Formatted at 2025-10-26 15:08:26 on dist-test-slave-8pgz"
server_key: "4db8452ca050839b1b975e2dedc5a06e"
server_key_iv: "ac8ac6e78785e3a49a65d10b718eeafb"
server_key_version: "encryptionkey@0"
I20251026 15:08:26.864645 21810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "03a2958b23f240b5b482844792cc6ed1"
format_stamp: "Formatted at 2025-10-26 15:08:26 on dist-test-slave-8pgz"
server_key: "4db8452ca050839b1b975e2dedc5a06e"
server_key_iv: "ac8ac6e78785e3a49a65d10b718eeafb"
server_key_version: "encryptionkey@0"
I20251026 15:08:26.865974 21810 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:08:26.866721 21825 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:26.866896 21810 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:26.866967 21810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "03a2958b23f240b5b482844792cc6ed1"
format_stamp: "Formatted at 2025-10-26 15:08:26 on dist-test-slave-8pgz"
server_key: "4db8452ca050839b1b975e2dedc5a06e"
server_key_iv: "ac8ac6e78785e3a49a65d10b718eeafb"
server_key_version: "encryptionkey@0"
I20251026 15:08:26.867055 21810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:26.897149 21810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:26.899493 21810 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:26.899633 21810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:26.899856 21810 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:26.900213 21810 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:26.900247 21810 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:26.900281 21810 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:26.900308 21810 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:26.906436 21810 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:37093
I20251026 15:08:26.906477 21938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:37093 every 8 connection(s)
I20251026 15:08:26.906783 21810 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:26.914502 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 21810
I20251026 15:08:26.914580 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:08:26.914733 15185 external_mini_cluster.cc:1468] Setting key 67926f068a7aa9b131bd7407c7ef8a44
I20251026 15:08:26.915766 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:26.907485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:37685 (local address 127.14.212.126:44653)
1026 15:08:26.907556 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:26.907559 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:26.907684 (+ 125us) server_negotiation.cc:408] Connection header received
1026 15:08:26.908228 (+ 544us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:26.908229 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:26.908249 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:26.908312 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:26.909109 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.909608 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:26.910653 (+ 1045us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.910831 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:26.912923 (+ 2092us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:26.912929 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:26.912929 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:26.912936 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:26.914272 (+ 1336us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.914709 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.914711 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.914711 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.914739 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.914999 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.915000 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.915001 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.915137 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:26.915209 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:26.915583 (+ 374us) server_negotiation.cc:300] Negotiation successful
1026 15:08:26.915675 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251026 15:08:26.915871 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:26.907418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:44653 (local address 127.14.212.67:37685)
1026 15:08:26.907583 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:26.907610 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:26.908125 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:26.908375 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:26.908378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:26.908717 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:26.909008 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:26.909011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.909697 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:26.909698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:26.910561 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:26.910564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.910714 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:26.911345 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:26.911363 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:26.912825 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:26.914360 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:26.914366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:26.914370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:26.914646 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:26.914826 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:26.914827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:26.914828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:26.914916 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:26.915201 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:26.915208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:26.915376 (+ 168us) client_negotiation.cc:769] Sending connection context
1026 15:08:26.915503 (+ 127us) client_negotiation.cc:240] Negotiation successful
1026 15:08:26.915674 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20251026 15:08:26.916373 21939 heartbeater.cc:344] Connected to a master server at 127.14.212.126:44653
I20251026 15:08:26.916460 21939 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:26.916643 21939 heartbeater.cc:507] Master 127.14.212.126:44653 requested a full tablet report, sending...
I20251026 15:08:26.917071 21474 ts_manager.cc:194] Registered new tserver with Master: 03a2958b23f240b5b482844792cc6ed1 (127.14.212.67:37093)
I20251026 15:08:26.917449 21474 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:37685
I20251026 15:08:26.926808 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/test-admin.keytab.
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:26.942488 15185 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20251026 15:08:26.944227 21946 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:26.943381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.14.212.65:38011)
1026 15:08:26.943534 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:26.943541 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:26.943551 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:26.943579 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:26.943581 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:26.943593 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:26.943663 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:26.944140 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55652: BlockingRecv error: recv got EOF from 127.0.0.1:55652 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
W20251026 15:08:26.944350 15185 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/krb5cc' of type FILE
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:26.945418 15185 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:26.946384 15185 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.65@KRBTEST.COM
I20251026 15:08:26.953806 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:26.947049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.14.212.65:38011)
1026 15:08:26.947107 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:26.947109 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:26.947117 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:08:26.947162 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:26.947163 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:26.947173 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:26.947216 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:26.947737 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.948451 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:26.948989 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.949177 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:26.951077 (+ 1900us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:26.951089 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:26.951089 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:26.951095 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:26.952782 (+ 1687us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.953126 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.953129 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.953129 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.953165 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.953302 (+ 137us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.953303 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.953304 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.953404 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:26.953558 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:26.953604 (+ 46us) server_negotiation.cc:300] Negotiation successful
1026 15:08:26.953688 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20251026 15:08:26.955557 21946 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:26.955071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.14.212.65:38011)
1026 15:08:26.955146 (+ 75us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:26.955148 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:26.955156 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:08:26.955175 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:26.955176 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:26.955184 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:26.955248 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:26.955500 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55672: BlockingRecv error: recv got EOF from 127.0.0.1:55672 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20251026 15:08:26.955698 15185 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/krb5cc' of type FILE
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:26.956687 15185 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:26.957543 15185 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.65@KRBTEST.COM
I20251026 15:08:26.964298 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:26.958100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.14.212.65:38011)
1026 15:08:26.958150 (+ 50us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:26.958152 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:26.958160 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:08:26.958262 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:26.958263 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:26.958271 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:26.958320 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:26.958845 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.959330 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:26.959839 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.960025 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:26.961772 (+ 1747us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:26.961775 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:26.961776 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:26.961781 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:26.963237 (+ 1456us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.963532 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.963534 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.963534 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.963558 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.963692 (+ 134us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.963693 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.963693 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.963827 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:26.963947 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:26.964078 (+ 131us) server_negotiation.cc:300] Negotiation successful
1026 15:08:26.964166 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20251026 15:08:26.965955 21946 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:26.965308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.14.212.65:38011)
1026 15:08:26.965352 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:26.965354 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:26.965448 (+ 94us) server_negotiation.cc:408] Connection header received
1026 15:08:26.965519 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:26.965520 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:26.965531 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:26.965588 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:26.965910 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55700: BlockingRecv error: recv got EOF from 127.0.0.1:55700 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20251026 15:08:26.966109 15185 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761491252308956-15185-0/krb5kdc/krb5cc' of type FILE
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:26.967129 15185 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:26.967936 15185 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 26 15:08:26 dist-test-slave-8pgz krb5kdc[21426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.65@KRBTEST.COM
I20251026 15:08:26.975087 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:26.968467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.14.212.65:38011)
1026 15:08:26.968516 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:26.968518 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:26.968525 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:08:26.968580 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:26.968581 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:26.968588 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:26.968633 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:26.969129 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.969578 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:26.970162 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:26.970305 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:26.972435 (+ 2130us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:26.972439 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:26.972439 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:26.972446 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:26.974058 (+ 1612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.974455 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.974456 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.974456 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.974480 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:26.974609 (+ 129us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:26.974609 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:26.974609 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:26.974678 (+ 69us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:26.974748 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:26.974941 (+ 193us) server_negotiation.cc:300] Negotiation successful
1026 15:08:26.975022 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251026 15:08:26.975786 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 21538
I20251026 15:08:26.980680 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 21674
I20251026 15:08:26.984975 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 21810
I20251026 15:08:26.989379 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 21442
2025-10-26T15:08:26Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2871 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:08:27 dist-test-slave-8pgz krb5kdc[21959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:08:27 dist-test-slave-8pgz krb5kdc[21959](info): set up 2 sockets
Oct 26 15:08:27 dist-test-slave-8pgz krb5kdc[21959](info): commencing operation
krb5kdc: starting...
W20251026 15:08:28.984478 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.975s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:08:29 dist-test-slave-8pgz krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:08:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
I20251026 15:08:29.125509 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:43071
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:43071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:29.206167 21975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:29.206384 21975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:29.206455 21975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:29.207875 21975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:29.207942 21975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:29.207965 21975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:29.207980 21975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:29.207994 21975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:08:29.209640 21975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:43071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:43071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:29.209865 21975 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:29.210076 21975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:29.212750 21981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:29.212755 21983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:29.212755 21980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:29.212872 21975 server_base.cc:1047] running on GCE node
I20251026 15:08:29.213188 21975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:29.213446 21975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:29.214584 21975 hybrid_clock.cc:648] HybridClock initialized: now 1761491309214551 us; error 49 us; skew 500 ppm
Oct 26 15:08:29 dist-test-slave-8pgz krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:29.216988 21975 init.cc:377] Logged in from keytab as oryx/127.14.212.126@KRBTEST.COM (short username oryx)
I20251026 15:08:29.217577 21975 webserver.cc:492] Webserver started at http://127.14.212.126:34123/ using document root <none> and password file <none>
I20251026 15:08:29.217768 21975 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:29.217808 21975 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:29.217933 21975 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:29.218904 21975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "44a9eda249f24e0cb8b00854b910ffb9"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "12095292fc9bfb2c7dbc2b7437e46cd1"
server_key_iv: "311c676c2ecf4d76c8cb96b2b4fb05c4"
server_key_version: "encryptionkey@0"
I20251026 15:08:29.219220 21975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "44a9eda249f24e0cb8b00854b910ffb9"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "12095292fc9bfb2c7dbc2b7437e46cd1"
server_key_iv: "311c676c2ecf4d76c8cb96b2b4fb05c4"
server_key_version: "encryptionkey@0"
I20251026 15:08:29.220407 21975 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:08:29.221078 21990 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:29.221261 21975 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:29.221339 21975 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "44a9eda249f24e0cb8b00854b910ffb9"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "12095292fc9bfb2c7dbc2b7437e46cd1"
server_key_iv: "311c676c2ecf4d76c8cb96b2b4fb05c4"
server_key_version: "encryptionkey@0"
I20251026 15:08:29.221401 21975 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:29.240409 21975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:29.247409 21975 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:29.247560 21975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:29.251484 21975 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:43071
I20251026 15:08:29.251528 22042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:43071 every 8 connection(s)
I20251026 15:08:29.251842 21975 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:29.252446 22043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:29.254963 22043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9: Bootstrap starting.
I20251026 15:08:29.255537 22043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:29.255788 22043 log.cc:826] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:29.256564 22043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9: No bootstrap required, opened a new log
I20251026 15:08:29.257738 22043 raft_consensus.cc:359] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } }
I20251026 15:08:29.257875 22043 raft_consensus.cc:385] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:29.257896 22043 raft_consensus.cc:740] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a9eda249f24e0cb8b00854b910ffb9, State: Initialized, Role: FOLLOWER
I20251026 15:08:29.258023 22043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } }
I20251026 15:08:29.258092 22043 raft_consensus.cc:399] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:29.258116 22043 raft_consensus.cc:493] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:29.258162 22043 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:29.258724 22043 raft_consensus.cc:515] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } }
I20251026 15:08:29.258821 22043 leader_election.cc:304] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44a9eda249f24e0cb8b00854b910ffb9; no voters:
I20251026 15:08:29.258975 22043 leader_election.cc:290] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:08:29.259018 22046 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:29.259150 22046 raft_consensus.cc:697] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [term 1 LEADER]: Becoming Leader. State: Replica: 44a9eda249f24e0cb8b00854b910ffb9, State: Running, Role: LEADER
I20251026 15:08:29.259207 22043 sys_catalog.cc:565] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:29.259284 22046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } }
I20251026 15:08:29.259666 22047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } }
I20251026 15:08:29.259707 22048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44a9eda249f24e0cb8b00854b910ffb9. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } }
I20251026 15:08:29.260270 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 21975
I20251026 15:08:29.260278 22047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:29.260301 22048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:29.260365 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:08:29.260478 22056 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:29.260515 15185 external_mini_cluster.cc:1468] Setting key 382378b8d6b1d1065796015e1dce46fb
I20251026 15:08:29.260910 22056 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:08:29.263005 22056 catalog_manager.cc:1357] Generated new cluster ID: 4b66c3f0fc7e41039c78c4f3971d7a0d
I20251026 15:08:29.263069 22056 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 26 15:08:29 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
I20251026 15:08:29.271535 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:29.261142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.14.212.126:43071)
1026 15:08:29.261308 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:29.261319 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:29.261567 (+ 248us) server_negotiation.cc:408] Connection header received
1026 15:08:29.261667 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:29.261672 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:29.261715 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:29.261821 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:29.262983 (+ 1162us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.264160 (+ 1177us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:29.265029 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.265654 (+ 625us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:29.267856 (+ 2202us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:29.267863 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:29.267864 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:29.267889 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:29.270079 (+ 2190us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:29.270487 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:29.270490 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:29.270490 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:29.270556 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:29.270712 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:29.270712 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:29.270713 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:29.270973 (+ 260us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:29.271115 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:29.271202 (+ 87us) server_negotiation.cc:300] Negotiation successful
1026 15:08:29.271346 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20251026 15:08:29.289418 22056 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:08:29.290220 22056 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:29.304541 22056 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9: Generated new TSK 0
I20251026 15:08:29.304764 22056 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
I20251026 15:08:29.356834 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:43071
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:29.437186 22071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:29.437398 22071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:29.437435 22071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:29.438930 22071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:29.438975 22071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:29.439028 22071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:08:29.440652 22071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:43071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:29.440896 22071 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:29.441089 22071 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:08:29.443867 22071 server_base.cc:1047] running on GCE node
W20251026 15:08:29.443830 22076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:29.443884 22077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:29.443830 22079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:29.444294 22071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:29.444487 22071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:29.445634 22071 hybrid_clock.cc:648] HybridClock initialized: now 1761491309445620 us; error 32 us; skew 500 ppm
Oct 26 15:08:29 dist-test-slave-8pgz krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:29.448325 22071 init.cc:377] Logged in from keytab as oryx/127.14.212.65@KRBTEST.COM (short username oryx)
I20251026 15:08:29.448879 22071 webserver.cc:492] Webserver started at http://127.14.212.65:44753/ using document root <none> and password file <none>
I20251026 15:08:29.449105 22071 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:29.449151 22071 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:29.449281 22071 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:29.450271 22071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "1f3d80d6732a4f28980d053270c2d095"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "9624124590d85444f4ffb06374458ffd"
server_key_iv: "a8527d1b933528954c01b50532b02f9b"
server_key_version: "encryptionkey@0"
I20251026 15:08:29.450632 22071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "1f3d80d6732a4f28980d053270c2d095"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "9624124590d85444f4ffb06374458ffd"
server_key_iv: "a8527d1b933528954c01b50532b02f9b"
server_key_version: "encryptionkey@0"
I20251026 15:08:29.451941 22071 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:29.452694 22086 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:29.452891 22071 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:08:29.453007 22071 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "1f3d80d6732a4f28980d053270c2d095"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "9624124590d85444f4ffb06374458ffd"
server_key_iv: "a8527d1b933528954c01b50532b02f9b"
server_key_version: "encryptionkey@0"
I20251026 15:08:29.453073 22071 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:29.472523 22071 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:29.474825 22071 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:29.474982 22071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:29.475250 22071 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:29.475572 22071 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:29.475605 22071 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:29.475642 22071 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:29.475675 22071 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:29.481524 22071 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:43321
I20251026 15:08:29.481588 22199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:43321 every 8 connection(s)
I20251026 15:08:29.481878 22071 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
Oct 26 15:08:29 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
I20251026 15:08:29.490340 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:29.482641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:46511 (local address 127.14.212.126:43071)
1026 15:08:29.482692 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:29.482695 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:29.482991 (+ 296us) server_negotiation.cc:408] Connection header received
1026 15:08:29.483528 (+ 537us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:29.483529 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:29.483546 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:29.483611 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:29.484390 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.484881 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:29.485413 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.485596 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:29.487942 (+ 2346us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:29.487945 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:29.487946 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:29.487953 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:29.489174 (+ 1221us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:29.489529 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:29.489530 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:29.489530 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:29.489555 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:29.489720 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:29.489721 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:29.489721 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:29.489811 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:29.489868 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:29.490184 (+ 316us) server_negotiation.cc:300] Negotiation successful
1026 15:08:29.490262 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251026 15:08:29.490499 22202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:29.482589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.65:46511)
1026 15:08:29.482849 (+ 260us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:29.482878 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:29.483426 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:29.483680 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:29.483684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:29.484008 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:29.484293 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:29.484296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.484961 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:29.484962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:29.485340 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:29.485341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.485468 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:29.486479 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:29.486495 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:29.487851 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:29.489260 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:29.489266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:29.489269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:29.489460 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:29.489611 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:29.489612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:29.489612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:29.489667 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:29.489898 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:29.489904 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:29.490046 (+ 142us) client_negotiation.cc:769] Sending connection context
1026 15:08:29.490154 (+ 108us) client_negotiation.cc:240] Negotiation successful
1026 15:08:29.490276 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
I20251026 15:08:29.491050 22200 heartbeater.cc:344] Connected to a master server at 127.14.212.126:43071
I20251026 15:08:29.491145 22200 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:29.491355 22200 heartbeater.cc:507] Master 127.14.212.126:43071 requested a full tablet report, sending...
I20251026 15:08:29.491714 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 22071
I20251026 15:08:29.491818 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:08:29.491941 22007 ts_manager.cc:194] Registered new tserver with Master: 1f3d80d6732a4f28980d053270c2d095 (127.14.212.65:43321)
I20251026 15:08:29.491976 15185 external_mini_cluster.cc:1468] Setting key bc0e386fbaf27e6eded59a495e6fa5d7
I20251026 15:08:29.492605 22007 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:46511
WARNING: no policy specified for oryx/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
I20251026 15:08:29.542739 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:43071
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:29.621500 22207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:29.621690 22207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:29.621718 22207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:29.623180 22207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:29.623230 22207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:29.623256 22207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:08:29.624856 22207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:43071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:29.625087 22207 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:29.625267 22207 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:29.627897 22213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:29.627933 22215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:29.628013 22207 server_base.cc:1047] running on GCE node
W20251026 15:08:29.627920 22212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:29.628242 22207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:29.628448 22207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:29.629603 22207 hybrid_clock.cc:648] HybridClock initialized: now 1761491309629560 us; error 61 us; skew 500 ppm
Oct 26 15:08:29 dist-test-slave-8pgz krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:29.632176 22207 init.cc:377] Logged in from keytab as oryx/127.14.212.66@KRBTEST.COM (short username oryx)
I20251026 15:08:29.632715 22207 webserver.cc:492] Webserver started at http://127.14.212.66:37849/ using document root <none> and password file <none>
I20251026 15:08:29.632900 22207 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:29.632941 22207 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:29.633061 22207 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:29.633973 22207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "47e0e140c7734c9b980eb8527b45eb9a"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "9545dfb8496e469d9f9996dcafc046a3"
server_key_iv: "a398b7a4c497fef6055ffbb8204a339c"
server_key_version: "encryptionkey@0"
I20251026 15:08:29.634292 22207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "47e0e140c7734c9b980eb8527b45eb9a"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "9545dfb8496e469d9f9996dcafc046a3"
server_key_iv: "a398b7a4c497fef6055ffbb8204a339c"
server_key_version: "encryptionkey@0"
I20251026 15:08:29.635507 22207 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:29.636232 22222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:29.636431 22207 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:29.636504 22207 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "47e0e140c7734c9b980eb8527b45eb9a"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "9545dfb8496e469d9f9996dcafc046a3"
server_key_iv: "a398b7a4c497fef6055ffbb8204a339c"
server_key_version: "encryptionkey@0"
I20251026 15:08:29.636564 22207 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:29.651778 22207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:29.653755 22207 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:29.653895 22207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:29.654145 22207 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:29.654552 22207 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:29.654588 22207 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:29.654626 22207 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:29.654656 22207 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:29.660418 22207 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:46657
I20251026 15:08:29.660496 22335 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:46657 every 8 connection(s)
I20251026 15:08:29.660789 22207 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
Oct 26 15:08:29 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
I20251026 15:08:29.667179 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 22207
I20251026 15:08:29.667302 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:08:29.667486 15185 external_mini_cluster.cc:1468] Setting key bf6ff59263446cb7b5b3bcf685ea6c89
I20251026 15:08:29.670096 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:29.661579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:47333 (local address 127.14.212.126:43071)
1026 15:08:29.661630 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:29.661632 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:29.661781 (+ 149us) server_negotiation.cc:408] Connection header received
1026 15:08:29.662316 (+ 535us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:29.662317 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:29.662327 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:29.662417 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:29.663289 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.663772 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:29.664375 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.664496 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:29.667081 (+ 2585us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:29.667087 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:29.667087 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:29.667096 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:29.668709 (+ 1613us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:29.669214 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:29.669215 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:29.669216 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:29.669238 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:29.669436 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:29.669436 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:29.669436 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:29.669548 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:29.669612 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:29.669880 (+ 268us) server_negotiation.cc:300] Negotiation successful
1026 15:08:29.669968 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20251026 15:08:29.670224 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:29.661516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.66:47333)
1026 15:08:29.661662 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:29.661708 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:29.662231 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:29.662515 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:29.662519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:29.662856 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:29.663194 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:29.663198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.663856 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:29.663856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:29.664288 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:29.664290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.664405 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:29.665403 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:29.665419 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:29.666937 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:29.668804 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:29.668810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:29.668813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:29.669131 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:29.669299 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:29.669300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:29.669300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:29.669383 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:29.669614 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:29.669621 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:29.669753 (+ 132us) client_negotiation.cc:769] Sending connection context
1026 15:08:29.669857 (+ 104us) client_negotiation.cc:240] Negotiation successful
1026 15:08:29.669997 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20251026 15:08:29.670717 22336 heartbeater.cc:344] Connected to a master server at 127.14.212.126:43071
I20251026 15:08:29.670817 22336 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:29.671008 22336 heartbeater.cc:507] Master 127.14.212.126:43071 requested a full tablet report, sending...
I20251026 15:08:29.671371 22007 ts_manager.cc:194] Registered new tserver with Master: 47e0e140c7734c9b980eb8527b45eb9a (127.14.212.66:46657)
I20251026 15:08:29.671753 22007 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:47333
WARNING: no policy specified for oryx/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
I20251026 15:08:29.711614 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:43071
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:29.793984 22343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:29.794157 22343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:29.794179 22343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:29.795590 22343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:29.795637 22343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:29.795663 22343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:08:29.797183 22343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:43071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:29.797354 22343 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:29.797497 22343 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:29.800120 22349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:29.800134 22348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:29.800134 22351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:29.800324 22343 server_base.cc:1047] running on GCE node
I20251026 15:08:29.800495 22343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:29.800705 22343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:29.801848 22343 hybrid_clock.cc:648] HybridClock initialized: now 1761491309801827 us; error 40 us; skew 500 ppm
Oct 26 15:08:29 dist-test-slave-8pgz krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:29.804430 22343 init.cc:377] Logged in from keytab as oryx/127.14.212.67@KRBTEST.COM (short username oryx)
I20251026 15:08:29.805008 22343 webserver.cc:492] Webserver started at http://127.14.212.67:34067/ using document root <none> and password file <none>
I20251026 15:08:29.805197 22343 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:29.805238 22343 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:29.805380 22343 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:29.806314 22343 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "acab8578ae284981b41ec9aa59a0b645"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "d3c0e37212f924dd164d0dfe8feb09d3"
server_key_iv: "fbfb8c3416f1fc3af149872cd2395921"
server_key_version: "encryptionkey@0"
I20251026 15:08:29.806654 22343 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "acab8578ae284981b41ec9aa59a0b645"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "d3c0e37212f924dd164d0dfe8feb09d3"
server_key_iv: "fbfb8c3416f1fc3af149872cd2395921"
server_key_version: "encryptionkey@0"
I20251026 15:08:29.807821 22343 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:29.808490 22358 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:29.808669 22343 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:29.808740 22343 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "acab8578ae284981b41ec9aa59a0b645"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "d3c0e37212f924dd164d0dfe8feb09d3"
server_key_iv: "fbfb8c3416f1fc3af149872cd2395921"
server_key_version: "encryptionkey@0"
I20251026 15:08:29.808800 22343 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:29.829643 22343 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:29.831856 22343 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:29.832000 22343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:29.832226 22343 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:29.832580 22343 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:29.832615 22343 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:29.832664 22343 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:29.832692 22343 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:29.838706 22343 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:46625
I20251026 15:08:29.838775 22471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:46625 every 8 connection(s)
I20251026 15:08:29.839119 22343 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
Oct 26 15:08:29 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
I20251026 15:08:29.846215 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 22343
I20251026 15:08:29.846310 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:08:29.846529 15185 external_mini_cluster.cc:1468] Setting key f9eac95838d30ef73c6727d4a5c123f9
I20251026 15:08:29.848008 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:29.839848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:59843 (local address 127.14.212.126:43071)
1026 15:08:29.839933 (+ 85us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:29.839935 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:29.840076 (+ 141us) server_negotiation.cc:408] Connection header received
1026 15:08:29.840601 (+ 525us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:29.840602 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:29.840616 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:29.840684 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:29.841547 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.842033 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:29.842617 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.842795 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:29.845208 (+ 2413us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:29.845212 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:29.845212 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:29.845220 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:29.846782 (+ 1562us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:29.847212 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:29.847213 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:29.847213 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:29.847235 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:29.847421 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:29.847422 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:29.847423 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:29.847519 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:29.847581 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:29.847830 (+ 249us) server_negotiation.cc:300] Negotiation successful
1026 15:08:29.847924 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251026 15:08:29.848168 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:29.839758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.67:59843)
1026 15:08:29.839930 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:29.839968 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:29.840505 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:29.840754 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:29.840757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:29.841135 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:29.841447 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:29.841450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.842116 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:29.842117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:29.842542 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:29.842544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:29.842679 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:29.843647 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:29.843664 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:29.845113 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:29.846915 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:29.846920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:29.846923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:29.847142 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:29.847300 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:29.847300 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:29.847301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:29.847359 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:29.847575 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:29.847579 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:29.847707 (+ 128us) client_negotiation.cc:769] Sending connection context
1026 15:08:29.847805 (+ 98us) client_negotiation.cc:240] Negotiation successful
1026 15:08:29.847943 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20251026 15:08:29.848613 22472 heartbeater.cc:344] Connected to a master server at 127.14.212.126:43071
I20251026 15:08:29.848702 22472 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:29.848873 22472 heartbeater.cc:507] Master 127.14.212.126:43071 requested a full tablet report, sending...
I20251026 15:08:29.849201 22007 ts_manager.cc:194] Registered new tserver with Master: acab8578ae284981b41ec9aa59a0b645 (127.14.212.67:46625)
I20251026 15:08:29.849584 22007 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:59843
I20251026 15:08:29.858352 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.212.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.212.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.212.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.212.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.212.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.212.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
I20251026 15:08:29.906232 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.125:33059
--webserver_interface=127.14.212.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:43071,127.14.212.125:33059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:29.990108 22479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:29.990304 22479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:29.990340 22479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:29.991808 22479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:29.991859 22479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:29.991884 22479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:29.991904 22479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:29.991925 22479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:08:29.993631 22479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:43071,127.14.212.125:33059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.125:33059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.212.125
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
W20251026 15:08:29.993744 22479 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.212.126:43071,127.14.212.125:33059'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20251026 15:08:29.993904 22479 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:29.994072 22479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:29.996688 22487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:29.996673 22485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:29.996757 22479 server_base.cc:1047] running on GCE node
W20251026 15:08:29.996672 22484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:29.997189 22479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:29.997444 22479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:29.998576 22479 hybrid_clock.cc:648] HybridClock initialized: now 1761491309998551 us; error 45 us; skew 500 ppm
Oct 26 15:08:29 dist-test-slave-8pgz krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:30.001088 22479 init.cc:377] Logged in from keytab as oryx/127.14.212.125@KRBTEST.COM (short username oryx)
I20251026 15:08:30.001633 22479 webserver.cc:492] Webserver started at http://127.14.212.125:43291/ using document root <none> and password file <none>
I20251026 15:08:30.001816 22479 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:30.001856 22479 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:30.001968 22479 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:30.002979 22479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/data/instance:
uuid: "6ac476a3eab7475199c6e4d1d67593d3"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "42b7f9a2316ee470725149c20f4bb8f8"
server_key_iv: "c62631ed091d81e9ac107bc80ecbce0e"
server_key_version: "encryptionkey@0"
I20251026 15:08:30.003307 22479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/wal/instance:
uuid: "6ac476a3eab7475199c6e4d1d67593d3"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "42b7f9a2316ee470725149c20f4bb8f8"
server_key_iv: "c62631ed091d81e9ac107bc80ecbce0e"
server_key_version: "encryptionkey@0"
I20251026 15:08:30.004523 22479 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:08:30.005218 22494 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:30.005496 22479 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:30.005571 22479 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/wal
uuid: "6ac476a3eab7475199c6e4d1d67593d3"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "42b7f9a2316ee470725149c20f4bb8f8"
server_key_iv: "c62631ed091d81e9ac107bc80ecbce0e"
server_key_version: "encryptionkey@0"
I20251026 15:08:30.005635 22479 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:30.018615 22479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:30.020612 22479 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:30.020745 22479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:30.024562 22479 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.125:33059
I20251026 15:08:30.024613 22546 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.125:33059 every 8 connection(s)
I20251026 15:08:30.024920 22479 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/data/info.pb
I20251026 15:08:30.025522 22547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:30.027344 22547 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:08:30.030690 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 22479
Oct 26 15:08:30 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.125@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
Oct 26 15:08:30 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.212.125@KRBTEST.COM
I20251026 15:08:30.036047 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.028046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.14.212.126:43071)
1026 15:08:30.028111 (+ 65us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.028113 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.028255 (+ 142us) server_negotiation.cc:408] Connection header received
1026 15:08:30.028593 (+ 338us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.028594 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.028604 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.028668 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.029373 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.029847 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.030445 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.030603 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.032905 (+ 2302us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.032909 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.032910 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.032918 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.034661 (+ 1743us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.035193 (+ 532us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.035194 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.035195 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.035215 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.035417 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.035417 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.035417 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.035525 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.035587 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.035853 (+ 266us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.035953 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251026 15:08:30.036162 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.027985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:48208)
1026 15:08:30.028139 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.028171 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.028524 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.028721 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.028724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.029017 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.029287 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.029290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.029912 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.029913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.030309 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.030311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.030442 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.031066 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.031084 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.032814 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.034744 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.034749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.034752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.035109 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.035267 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.035267 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.035268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.035354 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.035593 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.035598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.035725 (+ 127us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.035828 (+ 103us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.035955 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20251026 15:08:30.036751 22547 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:08:30.039391 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.031215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.14.212.125:33059)
1026 15:08:30.031373 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.031381 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.031400 (+ 19us) server_negotiation.cc:408] Connection header received
1026 15:08:30.031444 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.031446 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.031463 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.031530 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.032522 (+ 992us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.033316 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.034038 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.034269 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.036461 (+ 2192us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.036468 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.036469 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.036475 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.038311 (+ 1836us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.038644 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.038648 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.038649 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.038675 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.038834 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.038835 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.038835 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.038967 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.039122 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.039235 (+ 113us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.039320 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
Oct 26 15:08:30 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.125@KRBTEST.COM for oryx/127.14.212.125@KRBTEST.COM
I20251026 15:08:30.043013 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.037010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:33059 (local address 127.0.0.1:52766)
1026 15:08:30.037188 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.037198 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.037241 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.037383 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.037385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.037535 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.037657 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.037659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.038255 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.038256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.038606 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.038608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.038673 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.039113 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.039118 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.040532 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.042171 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.042172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.042172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.042298 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.042473 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.042474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.042474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.042500 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.042687 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.042687 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.042722 (+ 35us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.042783 (+ 61us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.042884 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20251026 15:08:30.043038 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.037059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.14.212.125:33059)
1026 15:08:30.037226 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.037228 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.037237 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:08:30.037300 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.037300 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.037308 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.037373 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.037725 (+ 352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.038188 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.038670 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.038843 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.040608 (+ 1765us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.040611 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.040611 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.040620 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.042086 (+ 1466us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.042359 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.042360 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.042360 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.042382 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.042550 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.042551 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.042551 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.042636 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.042694 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.042816 (+ 122us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.042948 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20251026 15:08:30.044555 22547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: Bootstrap starting.
I20251026 15:08:30.045166 22547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:30.045418 22547 log.cc:826] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:30.046180 22547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: No bootstrap required, opened a new log
I20251026 15:08:30.047344 22547 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } }
I20251026 15:08:30.047457 22547 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:30.047477 22547 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac476a3eab7475199c6e4d1d67593d3, State: Initialized, Role: FOLLOWER
I20251026 15:08:30.047564 22547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } }
I20251026 15:08:30.047843 22554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } } }
I20251026 15:08:30.047857 22547 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:30.047930 22554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: This master's current role is: FOLLOWER
W20251026 15:08:30.049508 22565 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:08:30.049569 22565 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:08:30.051023 22479 master_runner.cc:428] Detected that this master 6ac476a3eab7475199c6e4d1d67593d3 is joining an existing cluster
I20251026 15:08:30.051071 22479 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.212.125:33059
I20251026 15:08:30.057957 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.052076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.14.212.126:43071)
1026 15:08:30.052165 (+ 89us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.052167 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.052290 (+ 123us) server_negotiation.cc:408] Connection header received
1026 15:08:30.052390 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.052391 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.052402 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.052431 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.053296 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.053795 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.054355 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.054570 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.055362 (+ 792us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.055365 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.055366 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.055373 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.056985 (+ 1612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.057305 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.057306 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.057307 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.057333 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.057467 (+ 134us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.057467 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.057468 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.057556 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.057634 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.057772 (+ 138us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.057863 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251026 15:08:30.057981 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.052047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:48222)
1026 15:08:30.052219 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.052230 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.052313 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.052504 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.052505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.052658 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.053206 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.053208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.053889 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.053890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.054281 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.054283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.054349 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.054958 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.054962 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.055278 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.057084 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.057085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.057085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.057240 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.057385 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.057386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.057386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.057416 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.057625 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.057625 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.057671 (+ 46us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.057730 (+ 59us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.057861 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
Oct 26 15:08:30 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.126@KRBTEST.COM for oryx/127.14.212.125@KRBTEST.COM
I20251026 15:08:30.077855 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.060566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:33059 (local address 127.0.0.1:52784)
1026 15:08:30.060760 (+ 194us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.060787 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.060860 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.061008 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.061010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.061320 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.061483 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.061486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.062367 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.062368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.062752 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.062754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.062817 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.063285 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.063289 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.064854 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.076606 (+ 11752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.076619 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.076623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.076946 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.077162 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.077162 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.077163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.077200 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.077442 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.077449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.077510 (+ 61us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.077603 (+ 93us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.077725 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":97,"threads_started":1}
I20251026 15:08:30.077894 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.060655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.14.212.125:33059)
1026 15:08:30.060703 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.060705 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.060852 (+ 147us) server_negotiation.cc:408] Connection header received
1026 15:08:30.060933 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.060934 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.060945 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.061002 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.061563 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.062283 (+ 720us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.062833 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.063035 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.064939 (+ 1904us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.064943 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.064943 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.064950 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.076495 (+ 11545us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.077061 (+ 566us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.077062 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.077062 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.077089 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.077272 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.077273 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.077273 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.077372 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.077436 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.077639 (+ 203us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.077740 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20251026 15:08:30.078603 22007 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.14.212.125:33059: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3"
member_type: NON_VOTER
last_known_addr {
host: "127.14.212.125"
port: 33059
}
attrs {
promote: true
}
}
dest_uuid: "44a9eda249f24e0cb8b00854b910ffb9"
cas_config_opid_index: -1
I20251026 15:08:30.078912 22007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true } }
I20251026 15:08:30.079419 22574 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6ac476a3eab7475199c6e4d1d67593d3 (127.14.212.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true } } }
I20251026 15:08:30.079704 22521 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:30.080533 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.052097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:33059 (local address 127.0.0.1:52776)
1026 15:08:30.052237 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.052245 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.052313 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.052455 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.052455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.074632 (+ 22177us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.074940 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.074946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.075919 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.075922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.076552 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.076555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.076619 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.077082 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.077087 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.077433 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.079703 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.079703 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.079704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.079862 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.080000 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.080001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.080001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.080026 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.080242 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.080243 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.080273 (+ 30us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.080333 (+ 60us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.080423 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20251026 15:08:30.080559 22574 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.14.212.125:33059
I20251026 15:08:30.080579 22521 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 FOLLOWER]: Refusing update from remote peer 44a9eda249f24e0cb8b00854b910ffb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 15:08:30.080765 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.052121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.14.212.125:33059)
1026 15:08:30.052197 (+ 76us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.052199 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.052318 (+ 119us) server_negotiation.cc:408] Connection header received
1026 15:08:30.052370 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.052371 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.052382 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.052439 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.075050 (+ 22611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.075808 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.076766 (+ 958us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.076984 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.077490 (+ 506us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.077493 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.077493 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.077499 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.079611 (+ 2112us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.079926 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.079927 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.079927 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.079944 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.080075 (+ 131us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.080076 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.080076 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.080162 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.080215 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.080625 (+ 410us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.080694 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251026 15:08:30.080770 22479 master.cc:561] Master@127.14.212.125:33059 shutting down...
I20251026 15:08:30.081416 22576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true } } }
I20251026 15:08:30.081506 22576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:30.081645 22576 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 15:08:30.082299 21994 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20251026 15:08:30.082713 21994 consensus_peers.cc:597] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 -> Peer 6ac476a3eab7475199c6e4d1d67593d3 (127.14.212.125:33059): Couldn't send request to peer 6ac476a3eab7475199c6e4d1d67593d3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251026 15:08:30.083906 22479 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:30.083992 22479 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:30.084035 22479 tablet_replica.cc:333] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: stopping tablet replica
I20251026 15:08:30.088822 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.088644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:33059 (local address 127.0.0.1:52790)
1026 15:08:30.088743 (+ 99us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.088787 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20251026 15:08:30.096596 22479 master.cc:583] Master@127.14.212.125:33059 shutdown complete.
I20251026 15:08:30.096654 22479 master_runner.cc:305] Clearing existing system tablet
I20251026 15:08:30.096800 22479 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:30.096851 22479 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:30.097646 22479 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 15:08:30.098251 22580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:30.098450 22479 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:30.098500 22479 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251026 15:08:30.098524 22479 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/wal
uuid: "6ac476a3eab7475199c6e4d1d67593d3"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "42b7f9a2316ee470725149c20f4bb8f8"
server_key_iv: "c62631ed091d81e9ac107bc80ecbce0e"
server_key_version: "encryptionkey@0"
I20251026 15:08:30.099118 22479 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:30.100517 22479 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251026 15:08:30.100572 22479 log.cc:1199] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20251026 15:08:30.100745 22479 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: Deleting consensus metadata
I20251026 15:08:30.100852 22479 master_runner.cc:315] Copying system tablet from 127.14.212.126:43071
I20251026 15:08:30.101487 22479 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.126:43071
I20251026 15:08:30.107309 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.101858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.14.212.126:43071)
1026 15:08:30.101907 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.101909 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.101982 (+ 73us) server_negotiation.cc:408] Connection header received
1026 15:08:30.102052 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.102053 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.102062 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.102114 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.102773 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.103266 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.104047 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.104175 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.105024 (+ 849us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.105028 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.105028 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.105034 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.106434 (+ 1400us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.106783 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.106783 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.106783 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.106800 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.106927 (+ 127us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.106927 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.106928 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.107007 (+ 79us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.107053 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.107187 (+ 134us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.107245 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251026 15:08:30.107645 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.101773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:48236)
1026 15:08:30.101894 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.101906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.101986 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.102128 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.102129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.102332 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.102654 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.102658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.103356 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.103357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.103967 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.103969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.104047 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.104576 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.104581 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.104937 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.106515 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.106516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.106516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.106715 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.106848 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.106848 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.106849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.106877 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.107055 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.107056 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.107103 (+ 47us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.107173 (+ 70us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.107282 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20251026 15:08:30.108038 22027 tablet_copy_service.cc:140] P 44a9eda249f24e0cb8b00854b910ffb9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6ac476a3eab7475199c6e4d1d67593d3 ({username='oryx', principal='oryx/127.14.212.125@KRBTEST.COM'} at 127.0.0.1:48236)
I20251026 15:08:30.108135 22027 tablet_copy_service.cc:161] P 44a9eda249f24e0cb8b00854b910ffb9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6ac476a3eab7475199c6e4d1d67593d3 at {username='oryx', principal='oryx/127.14.212.125@KRBTEST.COM'} at 127.0.0.1:48236: session id = 6ac476a3eab7475199c6e4d1d67593d3-00000000000000000000000000000000
I20251026 15:08:30.108871 22027 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 15:08:30.109440 22479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251026 15:08:30.110507 22479 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: tablet copy: Starting download of 0 data blocks...
I20251026 15:08:30.110634 22479 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: tablet copy: Starting download of 1 WAL segments...
I20251026 15:08:30.111657 22479 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 15:08:30.112514 22027 tablet_copy_service.cc:342] P 44a9eda249f24e0cb8b00854b910ffb9: Request end of tablet copy session 6ac476a3eab7475199c6e4d1d67593d3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.212.125@KRBTEST.COM'} at 127.0.0.1:48236
I20251026 15:08:30.112578 22027 tablet_copy_service.cc:434] P 44a9eda249f24e0cb8b00854b910ffb9: ending tablet copy session 6ac476a3eab7475199c6e4d1d67593d3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6ac476a3eab7475199c6e4d1d67593d3
W20251026 15:08:30.114156 22479 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251026 15:08:30.115015 22479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:30.116243 22587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:30.116235 22590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:30.116349 22588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:30.116489 22479 server_base.cc:1047] running on GCE node
I20251026 15:08:30.116642 22479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:30.116794 22479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:30.117937 22479 hybrid_clock.cc:648] HybridClock initialized: now 1761491310117929 us; error 23 us; skew 500 ppm
Oct 26 15:08:30 dist-test-slave-8pgz krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491310, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:30.119653 22479 init.cc:377] Logged in from keytab as oryx/127.14.212.125@KRBTEST.COM (short username oryx)
I20251026 15:08:30.119951 22479 webserver.cc:492] Webserver started at http://127.14.212.125:35529/ using document root <none> and password file <none>
I20251026 15:08:30.120083 22479 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:30.120121 22479 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:30.120769 22479 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:30.121330 22597 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:30.121483 22479 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:30.121537 22479 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/wal
uuid: "6ac476a3eab7475199c6e4d1d67593d3"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "42b7f9a2316ee470725149c20f4bb8f8"
server_key_iv: "c62631ed091d81e9ac107bc80ecbce0e"
server_key_version: "encryptionkey@0"
I20251026 15:08:30.121595 22479 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:30.129981 22479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:30.131676 22479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:30.134701 22479 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.125:33059
I20251026 15:08:30.134770 22649 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.125:33059 every 8 connection(s)
I20251026 15:08:30.135039 22479 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-1/data/info.pb
I20251026 15:08:30.139106 22650 sys_catalog.cc:263] Verifying existing consensus state
I20251026 15:08:30.139659 22650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: Bootstrap starting.
I20251026 15:08:30.142607 22650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 15:08:30.142838 22650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: Bootstrap complete.
I20251026 15:08:30.143074 22650 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true } }
I20251026 15:08:30.143163 22650 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ac476a3eab7475199c6e4d1d67593d3, State: Initialized, Role: LEARNER
I20251026 15:08:30.143198 22650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true } }
I20251026 15:08:30.143368 22650 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:30.143663 22654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true } } }
I20251026 15:08:30.143829 22654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: This master's current role is: LEARNER
I20251026 15:08:30.145010 22665 catalog_manager.cc:1269] Loaded cluster ID: 4b66c3f0fc7e41039c78c4f3971d7a0d
I20251026 15:08:30.145059 22665 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: loading cluster ID for follower catalog manager: success
I20251026 15:08:30.146430 22665 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: acquiring CA information for follower catalog manager: success
I20251026 15:08:30.146714 22665 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 15:08:30.147784 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.141042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.14.212.125:33059)
1026 15:08:30.141195 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.141197 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.141231 (+ 34us) server_negotiation.cc:408] Connection header received
1026 15:08:30.141256 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.141258 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.141270 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.141341 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.141977 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.142736 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.143331 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.143575 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.144696 (+ 1121us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.144716 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.144717 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.144723 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.146681 (+ 1958us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.147019 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.147020 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.147021 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.147047 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.147266 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.147266 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.147267 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.147365 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.147467 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.147601 (+ 134us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.147698 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for oryx/127.14.212.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.212.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.212.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.212.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.212.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.212.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab.
I20251026 15:08:30.199258 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.124:41613
--webserver_interface=127.14.212.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:43071,127.14.212.125:33059,127.14.212.124:41613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:30.297238 22670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:30.297421 22670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:30.297443 22670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:30.298913 22670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:30.298964 22670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:30.298978 22670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:30.298990 22670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:30.299010 22670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:08:30.300594 22670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:43071,127.14.212.125:33059,127.14.212.124:41613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.124:41613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.212.124
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:30.300807 22670 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:30.300994 22670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:30.303534 22675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:30.303613 22678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:30.303591 22670 server_base.cc:1047] running on GCE node
W20251026 15:08:30.303557 22676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:30.303925 22670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:30.304185 22670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:30.305344 22670 hybrid_clock.cc:648] HybridClock initialized: now 1761491310305332 us; error 29 us; skew 500 ppm
Oct 26 15:08:30 dist-test-slave-8pgz krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491310, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:30.308071 22670 init.cc:377] Logged in from keytab as oryx/127.14.212.124@KRBTEST.COM (short username oryx)
I20251026 15:08:30.308607 22670 webserver.cc:492] Webserver started at http://127.14.212.124:37405/ using document root <none> and password file <none>
I20251026 15:08:30.308800 22670 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:30.308845 22670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:30.308972 22670 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:30.309924 22670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/data/instance:
uuid: "8c0b4a360db34b799231c882f33fbada"
format_stamp: "Formatted at 2025-10-26 15:08:30 on dist-test-slave-8pgz"
server_key: "4fc287922a85aa07a28dbe9004253360"
server_key_iv: "4b5d90316f041198178ebc1ea29f732d"
server_key_version: "encryptionkey@0"
I20251026 15:08:30.310268 22670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/wal/instance:
uuid: "8c0b4a360db34b799231c882f33fbada"
format_stamp: "Formatted at 2025-10-26 15:08:30 on dist-test-slave-8pgz"
server_key: "4fc287922a85aa07a28dbe9004253360"
server_key_iv: "4b5d90316f041198178ebc1ea29f732d"
server_key_version: "encryptionkey@0"
I20251026 15:08:30.311612 22670 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:30.312422 22685 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:30.312611 22670 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:30.312705 22670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/wal
uuid: "8c0b4a360db34b799231c882f33fbada"
format_stamp: "Formatted at 2025-10-26 15:08:30 on dist-test-slave-8pgz"
server_key: "4fc287922a85aa07a28dbe9004253360"
server_key_iv: "4b5d90316f041198178ebc1ea29f732d"
server_key_version: "encryptionkey@0"
I20251026 15:08:30.312772 22670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:30.338378 22670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:30.340528 22670 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:30.340679 22670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:30.344611 22670 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.124:41613
I20251026 15:08:30.344643 22737 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.124:41613 every 8 connection(s)
I20251026 15:08:30.345023 22670 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/data/info.pb
I20251026 15:08:30.345633 22738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:30.347411 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } has no permanent_uuid. Determining permanent_uuid...
Oct 26 15:08:30 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491310, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.124@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
I20251026 15:08:30.354547 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 22670
I20251026 15:08:30.355602 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.348159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.14.212.126:43071)
1026 15:08:30.348221 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.348223 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.348239 (+ 16us) server_negotiation.cc:408] Connection header received
1026 15:08:30.348608 (+ 369us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.348610 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.348622 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.348692 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.349447 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.349937 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.350606 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.350753 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.352753 (+ 2000us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.352758 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.352758 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.352766 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.354225 (+ 1459us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.354637 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.354639 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.354640 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.354670 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.354865 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.354865 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.354866 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.354960 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.355026 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.355393 (+ 367us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.355491 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251026 15:08:30.355619 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.348003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:48240)
1026 15:08:30.348153 (+ 150us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.348185 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.348516 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.348741 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.348745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.349071 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.349347 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.349350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.350066 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.350066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.350498 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.350500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.350591 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.351210 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.351229 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.352655 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.354311 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.354317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.354320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.354561 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.354750 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.354751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.354751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.354812 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.355035 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.355042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.355202 (+ 160us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.355339 (+ 137us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.355475 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20251026 15:08:30.356119 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } has no permanent_uuid. Determining permanent_uuid...
Oct 26 15:08:30 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491310, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.124@KRBTEST.COM for oryx/127.14.212.125@KRBTEST.COM
Oct 26 15:08:30 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.212.124@KRBTEST.COM
I20251026 15:08:30.362617 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.356443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.14.212.125:33059)
1026 15:08:30.356479 (+ 36us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.356482 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.356510 (+ 28us) server_negotiation.cc:408] Connection header received
1026 15:08:30.356534 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.356534 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.356543 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.356593 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.357019 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.357637 (+ 618us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.358271 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.358455 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.360039 (+ 1584us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.360045 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.360046 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.360053 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.361640 (+ 1587us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.361980 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.361981 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.361981 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.362006 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.362137 (+ 131us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.362137 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.362138 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.362223 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.362288 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.362426 (+ 138us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.362524 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20251026 15:08:30.362650 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.356379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:33059 (local address 127.0.0.1:52830)
1026 15:08:30.356426 (+ 47us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.356436 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.356477 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.356594 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.356594 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.356736 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.356943 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.356944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.357708 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.357709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.358189 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.358193 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.358263 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.358673 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.358679 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.359957 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.361715 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.361716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.361717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.361917 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.362060 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.362060 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.362061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.362087 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.362276 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.362276 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.362321 (+ 45us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.362380 (+ 59us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.362566 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20251026 15:08:30.363405 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:08:30.363914 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.355042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.14.212.124:41613)
1026 15:08:30.355215 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.355223 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.355236 (+ 13us) server_negotiation.cc:408] Connection header received
1026 15:08:30.355257 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.355260 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.355282 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.355342 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.356260 (+ 918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.357261 (+ 1001us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.357804 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.357970 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.360691 (+ 2721us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.360698 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.360699 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.360704 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.362631 (+ 1927us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.362960 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.362963 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.362963 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.363000 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.363176 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.363176 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.363177 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.363273 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.363404 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.363462 (+ 58us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.363539 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
Oct 26 15:08:30 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491310, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.124@KRBTEST.COM for oryx/127.14.212.124@KRBTEST.COM
I20251026 15:08:30.369428 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.363657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.0.0.1:56496)
1026 15:08:30.363689 (+ 32us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.363699 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.363737 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.363945 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.363946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.364083 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.364228 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.364229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.364802 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.364802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.365128 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.365129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.365191 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.365573 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.365576 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.366654 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.368710 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.368710 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.368711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.368838 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.369058 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.369058 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.369058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.369077 (+ 19us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.369230 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.369230 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.369259 (+ 29us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.369306 (+ 47us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.369372 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20251026 15:08:30.369570 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.363690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.14.212.124:41613)
1026 15:08:30.363859 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.363860 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.363867 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:08:30.363880 (+ 13us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.363880 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.363887 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.363937 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.364298 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.364738 (+ 440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.365193 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.365365 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.366730 (+ 1365us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.366732 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.366733 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.366738 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.368637 (+ 1899us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.368923 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.368924 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.368924 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.368951 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.369112 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.369113 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.369113 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.369187 (+ 74us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.369238 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.369320 (+ 82us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.369437 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":157,"thread_start_us":135,"threads_started":1}
I20251026 15:08:30.370782 22738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: Bootstrap starting.
I20251026 15:08:30.371373 22738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:30.371598 22738 log.cc:826] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:30.372267 22738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: No bootstrap required, opened a new log
I20251026 15:08:30.373354 22738 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } }
I20251026 15:08:30.373464 22738 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:30.373487 22738 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0b4a360db34b799231c882f33fbada, State: Initialized, Role: FOLLOWER
I20251026 15:08:30.373560 22738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } }
I20251026 15:08:30.373770 22744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } } }
I20251026 15:08:30.373785 22738 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:30.373867 22744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: This master's current role is: FOLLOWER
W20251026 15:08:30.375524 22755 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:08:30.375579 22755 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:08:30.377671 22670 master_runner.cc:428] Detected that this master 8c0b4a360db34b799231c882f33fbada is joining an existing cluster
I20251026 15:08:30.377707 22670 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.212.124:41613
I20251026 15:08:30.405839 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.378964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.14.212.126:43071)
1026 15:08:30.379016 (+ 52us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.379018 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.379027 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:08:30.379052 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.379053 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.379064 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.379119 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.399546 (+ 20427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.400331 (+ 785us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.401155 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.401350 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.402827 (+ 1477us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.402832 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.402832 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.402840 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.404506 (+ 1666us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.405060 (+ 554us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.405062 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.405062 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.405091 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.405258 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.405258 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.405259 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.405391 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.405435 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.405630 (+ 195us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.405718 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251026 15:08:30.405843 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.378857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.14.212.124:41613)
1026 15:08:30.378892 (+ 35us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.378894 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.381927 (+ 3033us) server_negotiation.cc:408] Connection header received
1026 15:08:30.381976 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.381977 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.381989 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.382041 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.399416 (+ 17375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.400121 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.400894 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.401080 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.402508 (+ 1428us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.402511 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.402511 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.402517 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.404589 (+ 2072us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.405063 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.405064 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.405065 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.405089 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.405289 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.405289 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.405290 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.405395 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.405445 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.405648 (+ 203us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.405731 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251026 15:08:30.406189 22762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.378607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:48250)
1026 15:08:30.378802 (+ 195us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.378813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.378874 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.379177 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.379178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.398881 (+ 19703us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.399387 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.399391 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.400453 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.400453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.401043 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.401045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.401410 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.402176 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.402182 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.402707 (+ 525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.404601 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.404602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.404603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.404957 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.405163 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.405164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.405164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.405194 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.405482 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.405483 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.405523 (+ 40us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.405996 (+ 473us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.406099 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":87,"threads_started":1}
I20251026 15:08:30.406937 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.378805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.0.0.1:56498)
1026 15:08:30.381842 (+ 3037us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.381852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.381920 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.382208 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.382209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.398887 (+ 16678us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.399269 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.399272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.400213 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.400213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.400802 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.400805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.401147 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.401833 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.401837 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.402417 (+ 580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.404689 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.404689 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.404690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.404965 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.405170 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.405170 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.405171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.405199 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.405497 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.405498 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.405532 (+ 34us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.406736 (+ 1204us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.406841 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3021,"thread_start_us":70,"threads_started":1}
I20251026 15:08:30.407438 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.378657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.14.212.125:33059)
1026 15:08:30.378696 (+ 39us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.378698 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.379137 (+ 439us) server_negotiation.cc:408] Connection header received
1026 15:08:30.379224 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.379225 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.379239 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.379296 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.399419 (+ 20123us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.400178 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.400901 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.401073 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.403974 (+ 2901us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.403978 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.403979 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.403989 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.406041 (+ 2052us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.406440 (+ 399us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.406442 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.406442 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.406467 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.406660 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.406661 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.406661 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.406762 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.406819 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.407050 (+ 231us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.407144 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20251026 15:08:30.407802 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.378614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:33059 (local address 127.0.0.1:52842)
1026 15:08:30.378931 (+ 317us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.378938 (+ 7us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.379134 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.379306 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.379307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.398845 (+ 19538us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.399266 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.399272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.400267 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.400267 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.400807 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.400808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.401147 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.401923 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.401927 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.403305 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.406157 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.406157 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.406158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.406330 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.406556 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.406557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.406558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.406591 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.406864 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.406865 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.406905 (+ 40us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.407573 (+ 668us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.407686 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":143,"threads_started":1}
Oct 26 15:08:30 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491309, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.126@KRBTEST.COM for oryx/127.14.212.124@KRBTEST.COM
I20251026 15:08:30.415014 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.408581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.14.212.124:41613)
1026 15:08:30.408620 (+ 39us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.408621 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.408635 (+ 14us) server_negotiation.cc:408] Connection header received
1026 15:08:30.408715 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.408716 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.408724 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.408771 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.409225 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.409625 (+ 400us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.410324 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.410556 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.412183 (+ 1627us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.412186 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.412187 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.412193 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.413918 (+ 1725us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.414235 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.414236 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.414236 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.414257 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.414385 (+ 128us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.414386 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.414386 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.414505 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.414575 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.414869 (+ 294us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.414939 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20251026 15:08:30.415081 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.408530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.0.0.1:56510)
1026 15:08:30.408594 (+ 64us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.408605 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.408647 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.408776 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.408777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.408942 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.409130 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.409132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.409704 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.409705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.410236 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.410237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.410319 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.410764 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.410769 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.412105 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.414031 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.414032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.414032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.414170 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.414317 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.414317 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.414317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.414339 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.414632 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.414633 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.414682 (+ 49us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.414762 (+ 80us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.414882 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20251026 15:08:30.415577 22007 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.14.212.124:41613: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8c0b4a360db34b799231c882f33fbada"
member_type: NON_VOTER
last_known_addr {
host: "127.14.212.124"
port: 41613
}
attrs {
promote: true
}
}
dest_uuid: "44a9eda249f24e0cb8b00854b910ffb9"
cas_config_opid_index: 5
I20251026 15:08:30.415822 22007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } }
I20251026 15:08:30.416188 22575 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8c0b4a360db34b799231c882f33fbada (127.14.212.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } } }
I20251026 15:08:30.417179 22575 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.14.212.124:41613
I20251026 15:08:30.417268 22575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } } }
I20251026 15:08:30.417318 22575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:30.417920 22670 master.cc:561] Master@127.14.212.124:41613 shutting down...
I20251026 15:08:30.417984 22712 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:30.418759 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.415568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.14.212.126:43071)
1026 15:08:30.415606 (+ 38us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.415608 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.415764 (+ 156us) server_negotiation.cc:408] Connection header received
1026 15:08:30.415808 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.415809 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.415819 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.415884 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:30.416156 (+ 272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.416824 (+ 668us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.418024 (+ 1200us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.418495 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.418563 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.418668 (+ 105us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.418701 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20251026 15:08:30.418897 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.415490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37178)
1026 15:08:30.415627 (+ 137us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.415636 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.415682 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.415902 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.415903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.415904 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:30.416071 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.416073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.416905 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.416906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.417924 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.417927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.418586 (+ 659us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.418589 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.418823 (+ 234us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.418854 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20251026 15:08:30.419076 22712 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 FOLLOWER]: Refusing update from remote peer 44a9eda249f24e0cb8b00854b910ffb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 15:08:30.419498 22576 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 15:08:30.420662 21991 consensus_peers.cc:597] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 -> Peer 8c0b4a360db34b799231c882f33fbada (127.14.212.124:41613): Couldn't send request to peer 8c0b4a360db34b799231c882f33fbada. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251026 15:08:30.421572 22670 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:30.421649 22670 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:30.421689 22670 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: stopping tablet replica
I20251026 15:08:30.422417 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.417513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:33059 (local address 127.0.0.1:58658)
1026 15:08:30.417660 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.417670 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.417717 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.420230 (+ 2513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.420231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.420233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:30.420428 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.420430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.421362 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.421363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.422160 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.422163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.422247 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.422251 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.422288 (+ 37us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.422336 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":104,"threads_started":1}
I20251026 15:08:30.423149 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.417586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.14.212.125:33059)
1026 15:08:30.420062 (+ 2476us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.420064 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.420072 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:08:30.420097 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.420098 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.420125 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.420179 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:30.420532 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.421256 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.422284 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.422767 (+ 483us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.423025 (+ 258us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.423050 (+ 25us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.423088 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2446}
I20251026 15:08:30.423377 22623 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 LEARNER]: Refusing update from remote peer 44a9eda249f24e0cb8b00854b910ffb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 15:08:30.423676 22576 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 15:08:30.424224 22623 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8c0b4a360db34b799231c882f33fbada (127.14.212.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } } }
I20251026 15:08:30.425231 22654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44a9eda249f24e0cb8b00854b910ffb9. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } } }
I20251026 15:08:30.425334 22654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: This master's current role is: LEARNER
I20251026 15:08:30.425387 22654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: NON_VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: true } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } } }
I20251026 15:08:30.425422 22576 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9: attempting to promote NON_VOTER 6ac476a3eab7475199c6e4d1d67593d3 to VOTER
I20251026 15:08:30.425434 22654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: This master's current role is: LEARNER
I20251026 15:08:30.425601 22576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } }
I20251026 15:08:30.426066 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.425951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.0.0.1:40874)
1026 15:08:30.425987 (+ 36us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.426040 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20251026 15:08:30.426100 22623 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 LEARNER]: Refusing update from remote peer 44a9eda249f24e0cb8b00854b910ffb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20251026 15:08:30.426497 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.426371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.0.0.1:40884)
1026 15:08:30.426438 (+ 67us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.426468 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20251026 15:08:30.426505 22575 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20251026 15:08:30.426663 21991 consensus_peers.cc:597] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 -> Peer 8c0b4a360db34b799231c882f33fbada (127.14.212.124:41613): Couldn't send request to peer 8c0b4a360db34b799231c882f33fbada. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41613: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 15:08:30.427130 22574 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6ac476a3eab7475199c6e4d1d67593d3 (127.14.212.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } } }
I20251026 15:08:30.428021 22574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } } }
I20251026 15:08:30.428094 22574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:30.428148 22574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } } }
I20251026 15:08:30.428184 22574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:30.428413 22623 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6ac476a3eab7475199c6e4d1d67593d3 (127.14.212.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } } }
I20251026 15:08:30.429191 22766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } } }
I20251026 15:08:30.429277 22766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:08:30.429672 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.429547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.0.0.1:40886)
1026 15:08:30.429618 (+ 71us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.429649 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20251026 15:08:30.429809 22006 master.cc:679] Network error: unable to get registration information for peer 8c0b4a360db34b799231c882f33fbada: Client connection negotiation failed: client connection to 127.14.212.124:41613: connect: Connection refused (error 111)
I20251026 15:08:30.434187 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.434068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.0.0.1:40894)
1026 15:08:30.434120 (+ 52us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.434156 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20251026 15:08:30.434366 22006 master.cc:679] Network error: unable to get registration information for peer 8c0b4a360db34b799231c882f33fbada: Client connection negotiation failed: client connection to 127.14.212.124:41613: connect: Connection refused (error 111)
I20251026 15:08:30.435051 22670 master.cc:583] Master@127.14.212.124:41613 shutdown complete.
I20251026 15:08:30.435103 22670 master_runner.cc:305] Clearing existing system tablet
I20251026 15:08:30.435256 22670 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:30.435305 22670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:30.436039 22670 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:30.436578 22768 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:30.436697 22670 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:30.436755 22670 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:30.436780 22670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/wal
uuid: "8c0b4a360db34b799231c882f33fbada"
format_stamp: "Formatted at 2025-10-26 15:08:30 on dist-test-slave-8pgz"
server_key: "4fc287922a85aa07a28dbe9004253360"
server_key_iv: "4b5d90316f041198178ebc1ea29f732d"
server_key_version: "encryptionkey@0"
I20251026 15:08:30.437503 22670 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:30.439043 22670 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251026 15:08:30.439116 22670 log.cc:1199] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20251026 15:08:30.439319 22670 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: Deleting consensus metadata
I20251026 15:08:30.439451 22670 master_runner.cc:315] Copying system tablet from 127.14.212.126:43071
I20251026 15:08:30.440065 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.439936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.0.0.1:40900)
1026 15:08:30.439974 (+ 38us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.440004 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20251026 15:08:30.440078 22670 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: tablet copy: Beginning tablet copy session from remote peer at address 127.14.212.126:43071
W20251026 15:08:30.440181 22006 master.cc:679] Network error: unable to get registration information for peer 8c0b4a360db34b799231c882f33fbada: Client connection negotiation failed: client connection to 127.14.212.124:41613: connect: Connection refused (error 111)
I20251026 15:08:30.446000 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.440385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37188)
1026 15:08:30.440532 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.440542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.440605 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.440738 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.440739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.440882 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:30.441059 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.441061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.441999 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.442001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.442460 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.442463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.442524 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.443009 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:30.443013 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:30.443338 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:30.445145 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.445146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.445147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.445319 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.445466 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:30.445467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:30.445467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:30.445491 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:30.445723 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:30.445724 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:30.445768 (+ 44us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.445839 (+ 71us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.445924 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20251026 15:08:30.446029 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.440440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.14.212.126:43071)
1026 15:08:30.440477 (+ 37us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.440479 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.440602 (+ 123us) server_negotiation.cc:408] Connection header received
1026 15:08:30.440665 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.440666 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.440676 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.440737 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:30.441118 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.441897 (+ 779us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.442542 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.442716 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.443412 (+ 696us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:30.443415 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:30.443415 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:30.443421 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:30.445051 (+ 1630us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.445386 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.445387 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.445388 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.445411 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:30.445535 (+ 124us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:30.445535 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:30.445536 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:30.445640 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:30.445705 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.445868 (+ 163us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.445946 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251026 15:08:30.446204 22027 tablet_copy_service.cc:140] P 44a9eda249f24e0cb8b00854b910ffb9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8c0b4a360db34b799231c882f33fbada ({username='oryx', principal='oryx/127.14.212.124@KRBTEST.COM'} at 127.0.0.1:37188)
I20251026 15:08:30.446240 22027 tablet_copy_service.cc:161] P 44a9eda249f24e0cb8b00854b910ffb9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8c0b4a360db34b799231c882f33fbada at {username='oryx', principal='oryx/127.14.212.124@KRBTEST.COM'} at 127.0.0.1:37188: session id = 8c0b4a360db34b799231c882f33fbada-00000000000000000000000000000000
I20251026 15:08:30.446835 22027 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 15:08:30.447350 22670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251026 15:08:30.448608 22670 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: tablet copy: Starting download of 0 data blocks...
I20251026 15:08:30.448743 22670 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: tablet copy: Starting download of 1 WAL segments...
I20251026 15:08:30.449517 22670 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 15:08:30.449769 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.449660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.0.0.1:40916)
1026 15:08:30.449713 (+ 53us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.449740 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20251026 15:08:30.449853 22006 master.cc:679] Network error: unable to get registration information for peer 8c0b4a360db34b799231c882f33fbada: Client connection negotiation failed: client connection to 127.14.212.124:41613: connect: Connection refused (error 111)
I20251026 15:08:30.450361 22027 tablet_copy_service.cc:342] P 44a9eda249f24e0cb8b00854b910ffb9: Request end of tablet copy session 8c0b4a360db34b799231c882f33fbada-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.212.124@KRBTEST.COM'} at 127.0.0.1:37188
I20251026 15:08:30.450438 22027 tablet_copy_service.cc:434] P 44a9eda249f24e0cb8b00854b910ffb9: ending tablet copy session 8c0b4a360db34b799231c882f33fbada-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8c0b4a360db34b799231c882f33fbada
W20251026 15:08:30.451889 22670 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251026 15:08:30.452677 22670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:30.453759 22776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:30.453819 22778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:30.453969 22775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:30.454041 22670 server_base.cc:1047] running on GCE node
I20251026 15:08:30.454128 22670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:30.454227 22670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:30.455338 22670 hybrid_clock.cc:648] HybridClock initialized: now 1761491310455306 us; error 48 us; skew 500 ppm
Oct 26 15:08:30 dist-test-slave-8pgz krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491310, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:30.456646 22670 init.cc:377] Logged in from keytab as oryx/127.14.212.124@KRBTEST.COM (short username oryx)
I20251026 15:08:30.456895 22670 webserver.cc:492] Webserver started at http://127.14.212.124:44781/ using document root <none> and password file <none>
I20251026 15:08:30.457013 22670 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:30.457080 22670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:30.457655 22670 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:30.458137 22785 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:30.458261 22670 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:30.458302 22670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/wal
uuid: "8c0b4a360db34b799231c882f33fbada"
format_stamp: "Formatted at 2025-10-26 15:08:30 on dist-test-slave-8pgz"
server_key: "4fc287922a85aa07a28dbe9004253360"
server_key_iv: "4b5d90316f041198178ebc1ea29f732d"
server_key_version: "encryptionkey@0"
I20251026 15:08:30.458355 22670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:30.467706 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.467569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.0.0.1:40928)
1026 15:08:30.467622 (+ 53us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.467660 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20251026 15:08:30.467875 22006 master.cc:679] Network error: unable to get registration information for peer 8c0b4a360db34b799231c882f33fbada: Client connection negotiation failed: client connection to 127.14.212.124:41613: connect: Connection refused (error 111)
I20251026 15:08:30.479506 22670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:30.481156 22670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:30.484164 22670 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.124:41613
I20251026 15:08:30.484277 22837 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.124:41613 every 8 connection(s)
I20251026 15:08:30.484519 22670 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/master-2/data/info.pb
I20251026 15:08:30.485195 22838 sys_catalog.cc:263] Verifying existing consensus state
I20251026 15:08:30.485570 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: Bootstrap starting.
I20251026 15:08:30.487824 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 15:08:30.488021 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: Bootstrap complete.
I20251026 15:08:30.488245 22838 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } }
I20251026 15:08:30.488319 22838 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c0b4a360db34b799231c882f33fbada, State: Initialized, Role: LEARNER
I20251026 15:08:30.488363 22838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } }
I20251026 15:08:30.488535 22838 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:30.488533 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } } }
I20251026 15:08:30.488624 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: This master's current role is: LEARNER
I20251026 15:08:30.489976 22852 catalog_manager.cc:1269] Loaded cluster ID: 4b66c3f0fc7e41039c78c4f3971d7a0d
I20251026 15:08:30.490027 22852 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: loading cluster ID for follower catalog manager: success
I20251026 15:08:30.491155 22852 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: acquiring CA information for follower catalog manager: success
I20251026 15:08:30.491415 22852 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 15:08:30.493415 22200 heartbeater.cc:499] Master 127.14.212.126:43071 was elected leader, sending a full tablet report...
I20251026 15:08:30.504439 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.501693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.0.0.1:40944)
1026 15:08:30.501747 (+ 54us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:30.501759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:30.501814 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:30.502066 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:30.502067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:30.502069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:30.502284 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.502287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.503616 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.503617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:30.504238 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:30.504240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.504313 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.504317 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:30.504348 (+ 31us) client_negotiation.cc:240] Negotiation successful
1026 15:08:30.504383 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20251026 15:08:30.505123 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:30.501778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.14.212.124:41613)
1026 15:08:30.501907 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:30.501910 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:30.501927 (+ 17us) server_negotiation.cc:408] Connection header received
1026 15:08:30.501953 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:30.501954 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:30.501975 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:30.502059 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:30.502360 (+ 301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.503511 (+ 1151us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:30.504327 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:30.504915 (+ 588us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:30.504991 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:30.505018 (+ 27us) server_negotiation.cc:300] Negotiation successful
1026 15:08:30.505059 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20251026 15:08:30.672564 22336 heartbeater.cc:499] Master 127.14.212.126:43071 was elected leader, sending a full tablet report...
I20251026 15:08:30.850411 22472 heartbeater.cc:499] Master 127.14.212.126:43071 was elected leader, sending a full tablet report...
I20251026 15:08:30.874008 22812 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20251026 15:08:30.874219 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44a9eda249f24e0cb8b00854b910ffb9. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } } }
I20251026 15:08:30.874332 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: This master's current role is: LEARNER
I20251026 15:08:31.278961 22576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: NON_VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: true } } }
I20251026 15:08:31.279137 22576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:31.279633 22854 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9: attempting to promote NON_VOTER 8c0b4a360db34b799231c882f33fbada to VOTER
I20251026 15:08:31.279811 22854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } }
I20251026 15:08:31.280303 22812 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 LEARNER]: Refusing update from remote peer 44a9eda249f24e0cb8b00854b910ffb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251026 15:08:31.280411 22623 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 FOLLOWER]: Refusing update from remote peer 44a9eda249f24e0cb8b00854b910ffb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251026 15:08:31.280584 22854 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251026 15:08:31.280730 22856 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251026 15:08:31.281487 22854 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8c0b4a360db34b799231c882f33fbada (127.14.212.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } } }
I20251026 15:08:31.281654 22623 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8c0b4a360db34b799231c882f33fbada (127.14.212.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } } }
I20251026 15:08:31.281687 22812 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8c0b4a360db34b799231c882f33fbada (127.14.212.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } } }
I20251026 15:08:31.282426 22854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } } }
I20251026 15:08:31.282516 22854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:31.282572 22854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } } }
I20251026 15:08:31.282613 22854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:31.282655 22854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } } }
I20251026 15:08:31.282691 22854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:31.282681 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } } }
I20251026 15:08:31.282758 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:08:31.282871 22858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44a9eda249f24e0cb8b00854b910ffb9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } } }
I20251026 15:08:31.282954 22858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:08:31.478564 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.476490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37200)
1026 15:08:31.476631 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.476642 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.476694 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.476832 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.476833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.476835 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:31.477002 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.477003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.477715 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.477716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.478354 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.478356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.478443 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.478448 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.478474 (+ 26us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.478502 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20251026 15:08:31.478971 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.476562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.14.212.126:43071)
1026 15:08:31.476675 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.476677 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.476684 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:08:31.476747 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.476748 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.476757 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.476813 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:31.477095 (+ 282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.477619 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.478459 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.478862 (+ 403us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.478869 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.478887 (+ 18us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.478914 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20251026 15:08:31.481308 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.479557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:33059 (local address 127.0.0.1:58668)
1026 15:08:31.479614 (+ 57us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.479627 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.479663 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.479846 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.479847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.479851 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:31.479964 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.479965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.480597 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.480597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.481143 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.481144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.481199 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.481203 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.481226 (+ 23us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.481256 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20251026 15:08:31.481714 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.479619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.14.212.125:33059)
1026 15:08:31.479750 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.479752 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.479758 (+ 6us) server_negotiation.cc:408] Connection header received
1026 15:08:31.479775 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.479776 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.479783 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.479834 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:31.480043 (+ 209us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.480536 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.481207 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.481622 (+ 415us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.481629 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.481647 (+ 18us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.481669 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20251026 15:08:31.484154 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.482413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.0.0.1:40946)
1026 15:08:31.482475 (+ 62us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.482487 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.482532 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.482722 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.482723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.482725 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:31.482833 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.482834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.483432 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.483433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.483944 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.483945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.483999 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.484001 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.484032 (+ 31us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.484093 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20251026 15:08:31.484827 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.482485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.14.212.124:41613)
1026 15:08:31.482619 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.482622 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.482628 (+ 6us) server_negotiation.cc:408] Connection header received
1026 15:08:31.482650 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.482651 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.482659 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.482714 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:31.482882 (+ 168us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.483377 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.484051 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.484697 (+ 646us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.484705 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.484728 (+ 23us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.484768 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20251026 15:08:31.492053 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.485820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.14.212.124:41613)
1026 15:08:31.485884 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.485886 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.485959 (+ 73us) server_negotiation.cc:408] Connection header received
1026 15:08:31.486068 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.486069 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.486077 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.486124 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:31.487110 (+ 986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.487579 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.488108 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.488315 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.489360 (+ 1045us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:31.489365 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:31.489365 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:31.489372 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:31.491075 (+ 1703us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.491387 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.491389 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.491389 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.491413 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.491560 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.491561 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.491561 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.491645 (+ 84us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:31.491739 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.491843 (+ 104us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.491937 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251026 15:08:31.494549 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.492527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37210)
1026 15:08:31.492667 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.492678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.492748 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.492872 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.492873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.492875 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:31.493048 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.493050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.493695 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.493696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.494291 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.494292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.494363 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.494369 (+ 6us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.494437 (+ 68us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.494475 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20251026 15:08:31.495031 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.492622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.14.212.126:43071)
1026 15:08:31.492683 (+ 61us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.492685 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.492732 (+ 47us) server_negotiation.cc:408] Connection header received
1026 15:08:31.492804 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.492805 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.492817 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.492866 (+ 49us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:31.493124 (+ 258us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.493622 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.494380 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.494931 (+ 551us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.494939 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.494958 (+ 19us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.494983 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251026 15:08:31.497493 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.495684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:33059 (local address 127.0.0.1:58670)
1026 15:08:31.495730 (+ 46us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.495740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.495832 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.495992 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.495993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.495996 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:31.496121 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.496122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.496758 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.496759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.497338 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.497339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.497399 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.497402 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.497425 (+ 23us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.497449 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20251026 15:08:31.497869 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.495758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.14.212.125:33059)
1026 15:08:31.495811 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.495812 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.495819 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:08:31.495925 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.495926 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.495934 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.495985 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:31.496212 (+ 227us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.496681 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.497406 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.497763 (+ 357us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.497770 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.497788 (+ 18us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.497810 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251026 15:08:31.500138 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.498518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.0.0.1:40966)
1026 15:08:31.498563 (+ 45us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.498575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.498619 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.498741 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.498742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.498743 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:31.498869 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.498870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.499432 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.499433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.499946 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.499947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.500012 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.500015 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.500048 (+ 33us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.500080 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20251026 15:08:31.500502 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.498570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40966 (local address 127.14.212.124:41613)
1026 15:08:31.498612 (+ 42us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.498614 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.498621 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:08:31.498678 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.498679 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.498687 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.498740 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:31.498924 (+ 184us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.499364 (+ 440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.500016 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.500426 (+ 410us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.500431 (+ 5us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.500449 (+ 18us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.500469 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20251026 15:08:31.501302 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 22071
I20251026 15:08:31.506536 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:43321
--local_ip_for_outbound_sockets=127.14.212.65
--tserver_master_addrs=127.14.212.126:43071
--webserver_port=44753
--webserver_interface=127.14.212.65
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.212.126:43071,127.14.212.125:33059
--tserver_master_addrs=127.14.212.126:43071,127.14.212.125:33059,127.14.212.124:41613 with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:31.586804 22865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:31.586997 22865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:31.587021 22865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:31.588474 22865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:31.588519 22865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:31.588546 22865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:08:31.590206 22865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:43321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=44753
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:43071,127.14.212.125:33059,127.14.212.124:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:31.590509 22865 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:31.590695 22865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:31.593325 22871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:31.593457 22870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:31.593485 22873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:31.593572 22865 server_base.cc:1047] running on GCE node
I20251026 15:08:31.593717 22865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:31.593919 22865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:31.595068 22865 hybrid_clock.cc:648] HybridClock initialized: now 1761491311595003 us; error 82 us; skew 500 ppm
Oct 26 15:08:31 dist-test-slave-8pgz krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491311, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:31.597792 22865 init.cc:377] Logged in from keytab as oryx/127.14.212.65@KRBTEST.COM (short username oryx)
I20251026 15:08:31.598421 22865 webserver.cc:492] Webserver started at http://127.14.212.65:44753/ using document root <none> and password file <none>
I20251026 15:08:31.598619 22865 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:31.598673 22865 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:31.599967 22865 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:31.600612 22880 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:31.600778 22865 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:31.600854 22865 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "1f3d80d6732a4f28980d053270c2d095"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "9624124590d85444f4ffb06374458ffd"
server_key_iv: "a8527d1b933528954c01b50532b02f9b"
server_key_version: "encryptionkey@0"
I20251026 15:08:31.601159 22865 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:31.619707 22865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:31.621915 22865 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:31.622090 22865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:31.622328 22865 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:31.622678 22865 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:31.622714 22865 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:31.622794 22865 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:31.622824 22865 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:31.628614 22865 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:43321
I20251026 15:08:31.628697 22993 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:43321 every 8 connection(s)
I20251026 15:08:31.629014 22865 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:08:31.631090 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 22865
I20251026 15:08:31.631285 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 22207
Oct 26 15:08:31 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491311, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
Oct 26 15:08:31 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491311, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.124@KRBTEST.COM
Oct 26 15:08:31 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491311, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.125@KRBTEST.COM
I20251026 15:08:31.640268 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:46657
--local_ip_for_outbound_sockets=127.14.212.66
--tserver_master_addrs=127.14.212.126:43071
--webserver_port=37849
--webserver_interface=127.14.212.66
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.212.126:43071,127.14.212.125:33059
--tserver_master_addrs=127.14.212.126:43071,127.14.212.125:33059,127.14.212.124:41613 with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:08:31.642314 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.630411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:48007 (local address 127.14.212.125:33059)
1026 15:08:31.630454 (+ 43us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.630457 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.630528 (+ 71us) server_negotiation.cc:408] Connection header received
1026 15:08:31.631014 (+ 486us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.631016 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.631029 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.631088 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:31.634034 (+ 2946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.634906 (+ 872us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.635725 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.635940 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.638725 (+ 2785us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:31.638731 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:31.638731 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:31.638739 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:31.640933 (+ 2194us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.641372 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.641374 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.641374 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.641400 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.641566 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.641566 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.641567 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.641703 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:31.641748 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.642100 (+ 352us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.642219 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251026 15:08:31.642326 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.630190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:39683 (local address 127.14.212.126:43071)
1026 15:08:31.630241 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.630264 (+ 23us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.630276 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:08:31.630824 (+ 548us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.630825 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.630838 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.630871 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:31.633065 (+ 2194us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.633770 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.634652 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.634834 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.638457 (+ 3623us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:31.638460 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:31.638461 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:31.638469 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:31.640629 (+ 2160us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.641268 (+ 639us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.641269 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.641269 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.641299 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.641560 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.641561 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.641561 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.641687 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:31.641732 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.642128 (+ 396us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.642236 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251026 15:08:31.642326 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.629769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:43139 (local address 127.14.212.124:41613)
1026 15:08:31.629833 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.629836 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.630019 (+ 183us) server_negotiation.cc:408] Connection header received
1026 15:08:31.630817 (+ 798us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.630818 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.630833 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.630896 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:31.632894 (+ 1998us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.633666 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.634667 (+ 1001us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.634846 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.638453 (+ 3607us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:31.638459 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:31.638459 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:31.638469 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:31.640558 (+ 2089us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.641248 (+ 690us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.641250 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.641251 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.641278 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.641561 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.641561 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.641562 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.641687 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:31.641732 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.642113 (+ 381us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.642220 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251026 15:08:31.642474 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.630320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:33059 (local address 127.14.212.65:48007)
1026 15:08:31.630452 (+ 132us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.630464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.630722 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.631146 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.631147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.633582 (+ 2435us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:31.633893 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.633896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.635035 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.635036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.635613 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.635625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.635707 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.636275 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:31.636280 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:31.638609 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:31.641032 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.641033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.641034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.641289 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.641466 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.641467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.641467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.641499 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.641791 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:31.641792 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:31.641940 (+ 148us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.642046 (+ 106us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.642223 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20251026 15:08:31.642721 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.630104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.65:39683)
1026 15:08:31.630221 (+ 117us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.630232 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.630706 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.630964 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.630968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.632282 (+ 1314us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:31.632960 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.632962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.633878 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.633879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.634548 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.634549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.634978 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.636041 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:31.636046 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:31.638310 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:31.640721 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.640732 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.640737 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.641181 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.641373 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.641373 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.641374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.641483 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.641783 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:31.641788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:31.641942 (+ 154us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.642439 (+ 497us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.642578 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20251026 15:08:31.642971 22996 heartbeater.cc:344] Connected to a master server at 127.14.212.125:33059
I20251026 15:08:31.643107 22996 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:31.643369 22996 heartbeater.cc:507] Master 127.14.212.125:33059 requested a full tablet report, sending...
I20251026 15:08:31.643515 22994 heartbeater.cc:344] Connected to a master server at 127.14.212.126:43071
I20251026 15:08:31.643572 22994 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:31.643683 22994 heartbeater.cc:507] Master 127.14.212.126:43071 requested a full tablet report, sending...
I20251026 15:08:31.643832 22614 ts_manager.cc:194] Registered new tserver with Master: 1f3d80d6732a4f28980d053270c2d095 (127.14.212.65:43321)
I20251026 15:08:31.643939 22006 ts_manager.cc:194] Re-registered known tserver with Master: 1f3d80d6732a4f28980d053270c2d095 (127.14.212.65:43321)
I20251026 15:08:31.644340 22006 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:39683
I20251026 15:08:31.644577 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.629747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.14.212.65:43139)
1026 15:08:31.629908 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.629935 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.630704 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.630964 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.630968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.632282 (+ 1314us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:31.632754 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.632760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.633878 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.633879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.634544 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.634547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.634978 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.635849 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:31.635878 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:31.638231 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:31.640881 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.640882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.640882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.641146 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.641373 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.641373 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.641374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.641483 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.641783 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:31.641788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:31.641948 (+ 160us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.643516 (+ 1568us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.643638 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20251026 15:08:31.644913 22995 heartbeater.cc:344] Connected to a master server at 127.14.212.124:41613
I20251026 15:08:31.644969 22995 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:31.645025 22995 heartbeater.cc:507] Master 127.14.212.124:41613 requested a full tablet report, sending...
I20251026 15:08:31.645349 22802 ts_manager.cc:194] Registered new tserver with Master: 1f3d80d6732a4f28980d053270c2d095 (127.14.212.65:43321)
W20251026 15:08:31.724797 23001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:31.724999 23001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:31.725033 23001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:31.726670 23001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:31.726719 23001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:31.726749 23001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:08:31.728514 23001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:46657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=37849
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:43071,127.14.212.125:33059,127.14.212.124:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:31.728760 23001 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:31.728950 23001 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:31.732343 23009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:31.732400 23001 server_base.cc:1047] running on GCE node
W20251026 15:08:31.732343 23006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:31.732331 23007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:31.732733 23001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:31.732918 23001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:31.734072 23001 hybrid_clock.cc:648] HybridClock initialized: now 1761491311734035 us; error 54 us; skew 500 ppm
Oct 26 15:08:31 dist-test-slave-8pgz krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491311, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:31.736814 23001 init.cc:377] Logged in from keytab as oryx/127.14.212.66@KRBTEST.COM (short username oryx)
I20251026 15:08:31.737437 23001 webserver.cc:492] Webserver started at http://127.14.212.66:37849/ using document root <none> and password file <none>
I20251026 15:08:31.737625 23001 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:31.737677 23001 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:31.739012 23001 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:31.739653 23016 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:31.739884 23001 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:31.739961 23001 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "47e0e140c7734c9b980eb8527b45eb9a"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "9545dfb8496e469d9f9996dcafc046a3"
server_key_iv: "a398b7a4c497fef6055ffbb8204a339c"
server_key_version: "encryptionkey@0"
I20251026 15:08:31.740248 23001 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:31.753643 23001 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:31.756083 23001 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:31.756222 23001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:31.756464 23001 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:31.756762 23001 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:31.756795 23001 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:31.756819 23001 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:31.756846 23001 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:31.762786 23001 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:46657
I20251026 15:08:31.762863 23129 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:46657 every 8 connection(s)
I20251026 15:08:31.763265 23001 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:08:31.766811 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 23001
I20251026 15:08:31.766963 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 22343
Oct 26 15:08:31 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491311, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.124@KRBTEST.COM
Oct 26 15:08:31 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491311, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
I20251026 15:08:31.773947 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.764461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:42835 (local address 127.14.212.126:43071)
1026 15:08:31.764504 (+ 43us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.764506 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.764514 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:08:31.764915 (+ 401us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.764917 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.764940 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.764977 (+ 37us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:31.766009 (+ 1032us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.766807 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.769231 (+ 2424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.769457 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.771294 (+ 1837us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:31.771298 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:31.771299 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:31.771306 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:31.772796 (+ 1490us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.773157 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.773158 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.773159 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.773184 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.773328 (+ 144us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.773329 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.773329 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.773429 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:31.773489 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.773756 (+ 267us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.773854 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251026 15:08:31.774129 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.764262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.66:42835)
1026 15:08:31.764401 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.764411 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.764805 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.765077 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.765077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.765476 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:31.765896 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.765901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.766952 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.766953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.769130 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.769133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.769209 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.769842 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:31.769847 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:31.771203 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:31.772890 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.772891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.772891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.773095 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.773243 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.773243 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.773244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.773273 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.773482 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:31.773489 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:31.773630 (+ 141us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.773732 (+ 102us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.773888 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20251026 15:08:31.774269 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.764209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:34057 (local address 127.14.212.124:41613)
1026 15:08:31.764333 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.764335 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.764343 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:08:31.764870 (+ 527us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.764872 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.764886 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.764947 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:31.766105 (+ 1158us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.766862 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.767996 (+ 1134us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.768204 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.770526 (+ 2322us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:31.770532 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:31.770533 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:31.770542 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:31.772559 (+ 2017us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.773066 (+ 507us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.773068 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.773068 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.773098 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.773346 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.773347 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.773347 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.773465 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:31.773507 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.774087 (+ 580us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.774182 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20251026 15:08:31.774559 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:46625
--local_ip_for_outbound_sockets=127.14.212.67
--tserver_master_addrs=127.14.212.126:43071
--webserver_port=34067
--webserver_interface=127.14.212.67
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.212.126:43071,127.14.212.125:33059
--tserver_master_addrs=127.14.212.126:43071,127.14.212.125:33059,127.14.212.124:41613 with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:08:31.774904 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.764020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.14.212.66:34057)
1026 15:08:31.764167 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.764202 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.764763 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.765017 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.765021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.765453 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:31.765979 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.765982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.766960 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.766961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.767893 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.767895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.768029 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.768683 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:31.768702 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:31.770347 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:31.772656 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.772663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.772667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.772988 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.773176 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.773177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.773177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.773274 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.773551 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:31.773551 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:31.773641 (+ 90us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.773732 (+ 91us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.774752 (+ 1020us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":104,"thread_start_us":60,"threads_started":1}
I20251026 15:08:31.775373 23131 heartbeater.cc:344] Connected to a master server at 127.14.212.124:41613
I20251026 15:08:31.775480 23131 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:31.775676 23131 heartbeater.cc:507] Master 127.14.212.124:41613 requested a full tablet report, sending...
I20251026 15:08:31.776090 23130 heartbeater.cc:344] Connected to a master server at 127.14.212.126:43071
I20251026 15:08:31.776098 22802 ts_manager.cc:194] Registered new tserver with Master: 47e0e140c7734c9b980eb8527b45eb9a (127.14.212.66:46657)
I20251026 15:08:31.776150 23130 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:31.776252 23130 heartbeater.cc:507] Master 127.14.212.126:43071 requested a full tablet report, sending...
I20251026 15:08:31.776484 22006 ts_manager.cc:194] Re-registered known tserver with Master: 47e0e140c7734c9b980eb8527b45eb9a (127.14.212.66:46657)
I20251026 15:08:31.777359 22006 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:42835
Oct 26 15:08:31 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491311, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.125@KRBTEST.COM
I20251026 15:08:31.781577 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.764159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:47349 (local address 127.14.212.125:33059)
1026 15:08:31.764253 (+ 94us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.764256 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.764297 (+ 41us) server_negotiation.cc:408] Connection header received
1026 15:08:31.764915 (+ 618us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.764917 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.764934 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.764968 (+ 34us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:31.766021 (+ 1053us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.766565 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.776800 (+ 10235us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.776988 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.778886 (+ 1898us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:31.778890 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:31.778890 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:31.778897 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:31.780461 (+ 1564us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.780940 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.780940 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.780941 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.780957 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.781132 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.781132 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.781133 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.781212 (+ 79us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:31.781258 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.781436 (+ 178us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.781514 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20251026 15:08:31.781702 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.764061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:33059 (local address 127.14.212.66:47349)
1026 15:08:31.764216 (+ 155us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.764227 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.764799 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.765065 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.765066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.765477 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:31.765896 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.765901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.775971 (+ 10070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.775973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.776709 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.776711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.776779 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.777266 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:31.777270 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:31.778783 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:31.780603 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.780604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.780605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.780834 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.781018 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.781019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.781019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.781067 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.781270 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:31.781271 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:31.781325 (+ 54us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.781412 (+ 87us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.781552 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20251026 15:08:31.782125 23132 heartbeater.cc:344] Connected to a master server at 127.14.212.125:33059
I20251026 15:08:31.782195 23132 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:31.782260 23132 heartbeater.cc:507] Master 127.14.212.125:33059 requested a full tablet report, sending...
I20251026 15:08:31.782567 22614 ts_manager.cc:194] Registered new tserver with Master: 47e0e140c7734c9b980eb8527b45eb9a (127.14.212.66:46657)
W20251026 15:08:31.858336 23137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:31.858565 23137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:31.858603 23137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:31.860213 23137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:31.860273 23137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:31.860316 23137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:08:31.861965 23137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:46625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=34067
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:43071,127.14.212.125:33059,127.14.212.124:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:31.862216 23137 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:31.862430 23137 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:31.864977 23143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:31.864977 23142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:31.864993 23145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:31.865178 23137 server_base.cc:1047] running on GCE node
I20251026 15:08:31.865375 23137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:31.865571 23137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:31.866695 23137 hybrid_clock.cc:648] HybridClock initialized: now 1761491311866682 us; error 25 us; skew 500 ppm
Oct 26 15:08:31 dist-test-slave-8pgz krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491311, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:31.869334 23137 init.cc:377] Logged in from keytab as oryx/127.14.212.67@KRBTEST.COM (short username oryx)
I20251026 15:08:31.869848 23137 webserver.cc:492] Webserver started at http://127.14.212.67:34067/ using document root <none> and password file <none>
I20251026 15:08:31.870031 23137 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:31.870083 23137 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:31.871444 23137 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:31.872068 23152 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:31.872280 23137 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:31.872363 23137 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "acab8578ae284981b41ec9aa59a0b645"
format_stamp: "Formatted at 2025-10-26 15:08:29 on dist-test-slave-8pgz"
server_key: "d3c0e37212f924dd164d0dfe8feb09d3"
server_key_iv: "fbfb8c3416f1fc3af149872cd2395921"
server_key_version: "encryptionkey@0"
I20251026 15:08:31.872617 23137 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:31.892673 23137 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:31.894788 23137 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:31.894928 23137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:31.895185 23137 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:31.895494 23137 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:31.895529 23137 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:31.895565 23137 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:31.895598 23137 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:31.901521 23137 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:46625
I20251026 15:08:31.901588 23265 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:46625 every 8 connection(s)
I20251026 15:08:31.901908 23137 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
Oct 26 15:08:31 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491311, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.125@KRBTEST.COM
Oct 26 15:08:31 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491311, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
Oct 26 15:08:31 dist-test-slave-8pgz krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491311, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.124@KRBTEST.COM
I20251026 15:08:31.910003 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 23137
I20251026 15:08:31.911974 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.902873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:36749 (local address 127.14.212.125:33059)
1026 15:08:31.902919 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.902921 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.902938 (+ 17us) server_negotiation.cc:408] Connection header received
1026 15:08:31.903586 (+ 648us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.903588 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.903600 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.903634 (+ 34us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:31.904855 (+ 1221us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.905484 (+ 629us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.906038 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.906198 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.908376 (+ 2178us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:31.908383 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:31.908383 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:31.908390 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:31.910420 (+ 2030us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.910934 (+ 514us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.910935 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.910936 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.910960 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.911192 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.911194 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.911194 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.911318 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:31.911760 (+ 442us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.911807 (+ 47us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.911894 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251026 15:08:31.912014 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.902625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:33059 (local address 127.14.212.67:36749)
1026 15:08:31.902829 (+ 204us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.902866 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.903475 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.903718 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.903719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.904110 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:31.904773 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.904775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.905559 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.905560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.905934 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.905936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.906248 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.906915 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:31.906918 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:31.908286 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:31.910520 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.910528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.910532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.910834 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.911037 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.911038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.911038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.911122 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.911384 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:31.911390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:31.911560 (+ 170us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.911673 (+ 113us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.911826 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20251026 15:08:31.912153 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.903095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:53113 (local address 127.14.212.126:43071)
1026 15:08:31.903140 (+ 45us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.903143 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.903152 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:08:31.903587 (+ 435us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.903588 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.903617 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.903648 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:31.904587 (+ 939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.905262 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.905961 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.906130 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.909344 (+ 3214us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:31.909349 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:31.909349 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:31.909357 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:31.910977 (+ 1620us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.911344 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.911346 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.911346 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.911375 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.911562 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.911563 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.911564 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.911681 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:31.911728 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.911948 (+ 220us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.912041 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251026 15:08:31.912492 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.902625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.67:53113)
1026 15:08:31.902829 (+ 204us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.902866 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.903468 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.903837 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.903837 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.904100 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:31.904464 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.904466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.905352 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.905352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.905877 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.905878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.906193 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.906755 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:31.906759 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:31.909215 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:31.911078 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.911080 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.911080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.911250 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.911456 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.911457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.911458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.911491 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.911778 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:31.911779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:31.911821 (+ 42us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.912259 (+ 438us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.912373 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20251026 15:08:31.912899 23266 heartbeater.cc:344] Connected to a master server at 127.14.212.126:43071
I20251026 15:08:31.912953 23268 heartbeater.cc:344] Connected to a master server at 127.14.212.125:33059
I20251026 15:08:31.912992 23268 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:31.912992 23266 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:31.913239 23266 heartbeater.cc:507] Master 127.14.212.126:43071 requested a full tablet report, sending...
I20251026 15:08:31.913250 23268 heartbeater.cc:507] Master 127.14.212.125:33059 requested a full tablet report, sending...
I20251026 15:08:31.913686 22614 ts_manager.cc:194] Registered new tserver with Master: acab8578ae284981b41ec9aa59a0b645 (127.14.212.67:46625)
I20251026 15:08:31.913851 22006 ts_manager.cc:194] Re-registered known tserver with Master: acab8578ae284981b41ec9aa59a0b645 (127.14.212.67:46625)
I20251026 15:08:31.914218 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.903131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:33343 (local address 127.14.212.124:41613)
1026 15:08:31.903177 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.903180 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.903205 (+ 25us) server_negotiation.cc:408] Connection header received
1026 15:08:31.903556 (+ 351us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.903557 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.903568 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.903626 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:31.904505 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.905163 (+ 658us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.905919 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.906099 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.909514 (+ 3415us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:31.909518 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:31.909518 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:31.909529 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:31.912586 (+ 3057us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.913299 (+ 713us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.913301 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.913301 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.913331 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.913556 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.913557 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.913558 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.913672 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:31.913740 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.914045 (+ 305us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.914134 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251026 15:08:31.914439 22006 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:53113
I20251026 15:08:31.914837 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.902877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41613 (local address 127.14.212.67:33343)
1026 15:08:31.903141 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.903152 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.903479 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.903700 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.903703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.904094 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:31.904409 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.904412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.905238 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.905239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.905835 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.905837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.905966 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.906649 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:31.906678 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:31.909398 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:31.912720 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.912721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.912722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.912909 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.913421 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:31.913421 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:31.913422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:31.913457 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:31.913809 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:31.913810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:31.913851 (+ 41us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.914602 (+ 751us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.914709 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":54,"threads_started":1}
I20251026 15:08:31.915182 23267 heartbeater.cc:344] Connected to a master server at 127.14.212.124:41613
I20251026 15:08:31.915236 23267 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:31.915303 23267 heartbeater.cc:507] Master 127.14.212.124:41613 requested a full tablet report, sending...
I20251026 15:08:31.915561 22802 ts_manager.cc:194] Registered new tserver with Master: acab8578ae284981b41ec9aa59a0b645 (127.14.212.67:46625)
I20251026 15:08:31.921021 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.913364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.14.212.126:43071)
1026 15:08:31.913399 (+ 35us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.913400 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.914108 (+ 708us) server_negotiation.cc:408] Connection header received
1026 15:08:31.914313 (+ 205us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.914314 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.914324 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.914352 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:31.915840 (+ 1488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.916612 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.917130 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.917322 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.918436 (+ 1114us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:31.918439 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:31.918440 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:31.918446 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:31.919898 (+ 1452us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.920272 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.920273 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.920274 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.920295 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.920436 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.920437 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.920437 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.920543 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:31.920654 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.920832 (+ 178us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.920916 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20251026 15:08:31.922497 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.913901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.14.212.124:41613)
1026 15:08:31.914294 (+ 393us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.914295 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.914713 (+ 418us) server_negotiation.cc:408] Connection header received
1026 15:08:31.914758 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.914758 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.914768 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.914830 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:31.916567 (+ 1737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.917252 (+ 685us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.918112 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.918318 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.919332 (+ 1014us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:31.919335 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:31.919336 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:31.919343 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:31.921480 (+ 2137us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.921805 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.921806 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.921806 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.921827 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.921969 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.921969 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.921969 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.922056 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:31.922178 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.922295 (+ 117us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.922368 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":54,"threads_started":1}
I20251026 15:08:31.922621 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.911617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.14.212.125:33059)
1026 15:08:31.912089 (+ 472us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.912091 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.913974 (+ 1883us) server_negotiation.cc:408] Connection header received
1026 15:08:31.914094 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.914095 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.914105 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.914135 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:31.915865 (+ 1730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.916323 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.917025 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.917181 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.919264 (+ 2083us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:31.919268 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:31.919269 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:31.919275 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:31.921118 (+ 1843us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.921447 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.921449 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.921449 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.921475 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:31.921666 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:31.921666 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:31.921667 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:31.921774 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:31.921882 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.922446 (+ 564us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.922536 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":62,"threads_started":1}
I20251026 15:08:31.922981 22006 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:08:31.924209 22006 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:08:31.930927 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.927966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:46625 (local address 127.0.0.1:58682)
1026 15:08:31.928518 (+ 552us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.928526 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.928575 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.928860 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.928861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.928863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:31.929043 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.929045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.929860 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.929862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.930705 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.930707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.930775 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.930780 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.930812 (+ 32us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.930852 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":76,"threads_started":1}
I20251026 15:08:31.931244 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.927949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46657 (local address 127.0.0.1:35030)
1026 15:08:31.928355 (+ 406us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.928366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.928433 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.928711 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.928713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.928715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:31.928957 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.928960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.930201 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.930202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.931051 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.931053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.931113 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.931117 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.931146 (+ 29us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.931186 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":95,"threads_started":1}
I20251026 15:08:31.931908 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.928309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.14.212.67:46625)
1026 15:08:31.928457 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.928464 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.928593 (+ 129us) server_negotiation.cc:408] Connection header received
1026 15:08:31.928723 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.928726 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.928745 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.928854 (+ 109us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:31.929103 (+ 249us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.929781 (+ 678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.931127 (+ 1346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.931705 (+ 578us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.931764 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.931796 (+ 32us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.931830 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20251026 15:08:31.931998 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.928057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:43321 (local address 127.0.0.1:56824)
1026 15:08:31.928644 (+ 587us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:31.928651 (+ 7us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:31.928738 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:31.928998 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:31.928999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:31.929001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:31.929222 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.929224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.930499 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.930500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:31.931302 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:31.931306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.931824 (+ 518us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.931828 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:31.931894 (+ 66us) client_negotiation.cc:240] Negotiation successful
1026 15:08:31.931934 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":86,"threads_started":1}
I20251026 15:08:31.932029 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.928144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.14.212.65:43321)
1026 15:08:31.928286 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.928295 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.928708 (+ 413us) server_negotiation.cc:408] Connection header received
1026 15:08:31.928831 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.928834 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.928851 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.928894 (+ 43us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:31.929312 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.930362 (+ 1050us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.931415 (+ 1053us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.931763 (+ 348us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.931800 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.931920 (+ 120us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.931956 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20251026 15:08:31.932423 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:31.928165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.14.212.66:46657)
1026 15:08:31.928331 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:31.928340 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:31.928417 (+ 77us) server_negotiation.cc:408] Connection header received
1026 15:08:31.928516 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:31.928519 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:31.928538 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:31.928607 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:31.929028 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.930121 (+ 1093us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:31.931440 (+ 1319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:31.931922 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:31.931965 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:31.932299 (+ 334us) server_negotiation.cc:300] Negotiation successful
1026 15:08:31.932351 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
I20251026 15:08:31.932511 23200 tablet_service.cc:1505] Processing CreateTablet for tablet ef4d29a2974c405fb023ed2a7667604c (DEFAULT_TABLE table=test-table [id=ae31372a76e54be4afba2715cdf515bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:31.932566 22928 tablet_service.cc:1505] Processing CreateTablet for tablet ef4d29a2974c405fb023ed2a7667604c (DEFAULT_TABLE table=test-table [id=ae31372a76e54be4afba2715cdf515bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:31.932796 22928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef4d29a2974c405fb023ed2a7667604c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:31.932819 23200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef4d29a2974c405fb023ed2a7667604c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:31.934883 23064 tablet_service.cc:1505] Processing CreateTablet for tablet ef4d29a2974c405fb023ed2a7667604c (DEFAULT_TABLE table=test-table [id=ae31372a76e54be4afba2715cdf515bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:31.935168 23064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef4d29a2974c405fb023ed2a7667604c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:31.935179 23296 tablet_bootstrap.cc:492] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095: Bootstrap starting.
I20251026 15:08:31.935348 23297 tablet_bootstrap.cc:492] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645: Bootstrap starting.
I20251026 15:08:31.936043 23296 tablet_bootstrap.cc:654] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:31.936229 23297 tablet_bootstrap.cc:654] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:31.936347 23296 log.cc:826] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:31.936504 23297 log.cc:826] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:31.937305 23297 tablet_bootstrap.cc:492] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645: No bootstrap required, opened a new log
I20251026 15:08:31.937368 23297 ts_tablet_manager.cc:1403] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:08:31.937440 23296 tablet_bootstrap.cc:492] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095: No bootstrap required, opened a new log
I20251026 15:08:31.937494 23296 ts_tablet_manager.cc:1403] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251026 15:08:31.939008 23296 raft_consensus.cc:359] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:31.939126 23296 raft_consensus.cc:385] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:31.939145 23296 raft_consensus.cc:740] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f3d80d6732a4f28980d053270c2d095, State: Initialized, Role: FOLLOWER
I20251026 15:08:31.939220 23296 consensus_queue.cc:260] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:31.939486 22994 heartbeater.cc:499] Master 127.14.212.126:43071 was elected leader, sending a full tablet report...
I20251026 15:08:31.939442 23297 raft_consensus.cc:359] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:31.939584 23297 raft_consensus.cc:385] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:31.939618 23297 raft_consensus.cc:740] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acab8578ae284981b41ec9aa59a0b645, State: Initialized, Role: FOLLOWER
I20251026 15:08:31.939713 23297 consensus_queue.cc:260] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:31.939961 23297 ts_tablet_manager.cc:1434] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251026 15:08:31.940068 23266 heartbeater.cc:499] Master 127.14.212.126:43071 was elected leader, sending a full tablet report...
I20251026 15:08:31.940207 23296 ts_tablet_manager.cc:1434] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 15:08:31.941511 23300 tablet_bootstrap.cc:492] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a: Bootstrap starting.
I20251026 15:08:31.942425 23300 tablet_bootstrap.cc:654] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:31.942759 23300 log.cc:826] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:31.943627 23300 tablet_bootstrap.cc:492] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a: No bootstrap required, opened a new log
I20251026 15:08:31.943701 23300 ts_tablet_manager.cc:1403] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:08:31.945483 23300 raft_consensus.cc:359] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:31.945633 23300 raft_consensus.cc:385] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:31.945663 23300 raft_consensus.cc:740] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e0e140c7734c9b980eb8527b45eb9a, State: Initialized, Role: FOLLOWER
I20251026 15:08:31.945770 23300 consensus_queue.cc:260] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:31.946017 23300 ts_tablet_manager.cc:1434] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251026 15:08:31.946089 23130 heartbeater.cc:499] Master 127.14.212.126:43071 was elected leader, sending a full tablet report...
W20251026 15:08:32.013903 23133 tablet.cc:2378] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:08:32.032943 23302 raft_consensus.cc:493] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:08:32.033128 23302 raft_consensus.cc:515] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:32.033473 23302 leader_election.cc:290] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47e0e140c7734c9b980eb8527b45eb9a (127.14.212.66:46657), 1f3d80d6732a4f28980d053270c2d095 (127.14.212.65:43321)
I20251026 15:08:32.036522 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:32.033702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:43321 (local address 127.14.212.67:52371)
1026 15:08:32.033752 (+ 50us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:32.033761 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:32.033802 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:32.034040 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:32.034042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:32.034045 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:32.034317 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:32.034319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.035333 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:32.035334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:32.036265 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:32.036267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.036359 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:32.036363 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:32.036400 (+ 37us) client_negotiation.cc:240] Negotiation successful
1026 15:08:32.036436 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20251026 15:08:32.036522 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:32.033686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46657 (local address 127.14.212.67:35127)
1026 15:08:32.033743 (+ 57us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:32.033757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:32.033802 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:32.034040 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:32.034042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:32.034045 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:32.034317 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:32.034319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.035315 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:32.035317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:32.036298 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:32.036300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.036359 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:32.036363 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:32.036400 (+ 37us) client_negotiation.cc:240] Negotiation successful
1026 15:08:32.036436 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20251026 15:08:32.036996 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:32.033819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:52371 (local address 127.14.212.65:43321)
1026 15:08:32.033867 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:32.033870 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:32.033880 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:32.033911 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:32.033912 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:32.033921 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:32.033996 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:32.034434 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.035255 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:32.036381 (+ 1126us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.036858 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:32.036866 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:32.036889 (+ 23us) server_negotiation.cc:300] Negotiation successful
1026 15:08:32.036918 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251026 15:08:32.036986 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:32.033820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:35127 (local address 127.14.212.66:46657)
1026 15:08:32.033867 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:32.033870 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:32.033885 (+ 15us) server_negotiation.cc:408] Connection header received
1026 15:08:32.033911 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:32.033912 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:32.033921 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:32.033988 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:32.034434 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.035207 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:32.036381 (+ 1174us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.036857 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:32.036866 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:32.036889 (+ 23us) server_negotiation.cc:300] Negotiation successful
1026 15:08:32.036920 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251026 15:08:32.037329 23084 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef4d29a2974c405fb023ed2a7667604c" candidate_uuid: "acab8578ae284981b41ec9aa59a0b645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e0e140c7734c9b980eb8527b45eb9a" is_pre_election: true
I20251026 15:08:32.037387 22948 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef4d29a2974c405fb023ed2a7667604c" candidate_uuid: "acab8578ae284981b41ec9aa59a0b645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f3d80d6732a4f28980d053270c2d095" is_pre_election: true
I20251026 15:08:32.037460 23084 raft_consensus.cc:2468] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acab8578ae284981b41ec9aa59a0b645 in term 0.
I20251026 15:08:32.037539 22948 raft_consensus.cc:2468] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acab8578ae284981b41ec9aa59a0b645 in term 0.
I20251026 15:08:32.037705 23153 leader_election.cc:304] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e0e140c7734c9b980eb8527b45eb9a, acab8578ae284981b41ec9aa59a0b645; no voters:
I20251026 15:08:32.037830 23302 raft_consensus.cc:2804] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:08:32.037907 23302 raft_consensus.cc:493] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:08:32.037940 23302 raft_consensus.cc:3060] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:32.038697 23302 raft_consensus.cc:515] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:32.038892 23302 leader_election.cc:290] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [CANDIDATE]: Term 1 election: Requested vote from peers 47e0e140c7734c9b980eb8527b45eb9a (127.14.212.66:46657), 1f3d80d6732a4f28980d053270c2d095 (127.14.212.65:43321)
I20251026 15:08:32.038975 23084 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef4d29a2974c405fb023ed2a7667604c" candidate_uuid: "acab8578ae284981b41ec9aa59a0b645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e0e140c7734c9b980eb8527b45eb9a"
I20251026 15:08:32.039019 22948 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef4d29a2974c405fb023ed2a7667604c" candidate_uuid: "acab8578ae284981b41ec9aa59a0b645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f3d80d6732a4f28980d053270c2d095"
I20251026 15:08:32.039064 23084 raft_consensus.cc:3060] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:32.039083 22948 raft_consensus.cc:3060] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:32.039937 22948 raft_consensus.cc:2468] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acab8578ae284981b41ec9aa59a0b645 in term 1.
I20251026 15:08:32.039937 23084 raft_consensus.cc:2468] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acab8578ae284981b41ec9aa59a0b645 in term 1.
I20251026 15:08:32.040159 23154 leader_election.cc:304] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f3d80d6732a4f28980d053270c2d095, acab8578ae284981b41ec9aa59a0b645; no voters:
I20251026 15:08:32.040299 23302 raft_consensus.cc:2804] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:32.040589 23302 raft_consensus.cc:697] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [term 1 LEADER]: Becoming Leader. State: Replica: acab8578ae284981b41ec9aa59a0b645, State: Running, Role: LEADER
I20251026 15:08:32.040797 23302 consensus_queue.cc:237] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:32.041618 22006 catalog_manager.cc:5649] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 reported cstate change: term changed from 0 to 1, leader changed from <none> to acab8578ae284981b41ec9aa59a0b645 (127.14.212.67). New cstate: current_term: 1 leader_uuid: "acab8578ae284981b41ec9aa59a0b645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } health_report { overall_health: UNKNOWN } } }
I20251026 15:08:32.050848 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:32.048723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.14.212.67:46625)
1026 15:08:32.048800 (+ 77us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:32.048802 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:32.048811 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:08:32.048850 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:32.048851 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:32.048874 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:32.048936 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:08:32.049161 (+ 225us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.049643 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:32.050175 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.050312 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:32.050347 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:08:32.050496 (+ 149us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:08:32.050562 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:32.050701 (+ 139us) server_negotiation.cc:300] Negotiation successful
1026 15:08:32.050741 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251026 15:08:32.052546 22948 raft_consensus.cc:1275] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095 [term 1 FOLLOWER]: Refusing update from remote peer acab8578ae284981b41ec9aa59a0b645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:32.052546 23084 raft_consensus.cc:1275] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a [term 1 FOLLOWER]: Refusing update from remote peer acab8578ae284981b41ec9aa59a0b645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:32.052871 23302 consensus_queue.cc:1048] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:32.052990 23305 consensus_queue.cc:1048] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:32.055616 23310 mvcc.cc:204] Tried to move back new op lower bound from 7215068414164676608 to 7215068414119731200. Current Snapshot: MvccSnapshot[applied={T|T < 7215068414164676608}]
I20251026 15:08:32.057476 23309 mvcc.cc:204] Tried to move back new op lower bound from 7215068414164676608 to 7215068414119731200. Current Snapshot: MvccSnapshot[applied={T|T < 7215068414164676608}]
I20251026 15:08:32.057761 22006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37224:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:32.057869 22006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37224:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:32.059199 22006 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 44a9eda249f24e0cb8b00854b910ffb9: Sending DeleteTablet for 3 replicas of tablet ef4d29a2974c405fb023ed2a7667604c
I20251026 15:08:32.059444 23064 tablet_service.cc:1552] Processing DeleteTablet for tablet ef4d29a2974c405fb023ed2a7667604c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:32 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:35030
I20251026 15:08:32.059602 23200 tablet_service.cc:1552] Processing DeleteTablet for tablet ef4d29a2974c405fb023ed2a7667604c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:32 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:58682
I20251026 15:08:32.059849 23317 tablet_replica.cc:333] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a: stopping tablet replica
I20251026 15:08:32.059852 23318 tablet_replica.cc:333] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645: stopping tablet replica
I20251026 15:08:32.059890 22928 tablet_service.cc:1552] Processing DeleteTablet for tablet ef4d29a2974c405fb023ed2a7667604c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:32 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:56824
I20251026 15:08:32.059952 23317 raft_consensus.cc:2243] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:32.060020 23317 raft_consensus.cc:2272] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:32.060111 23318 raft_consensus.cc:2243] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:08:32.060248 23318 raft_consensus.cc:2272] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:32.060464 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 21975
I20251026 15:08:32.061750 23318 ts_tablet_manager.cc:1916] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:32.062520 23319 tablet_replica.cc:333] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095: stopping tablet replica
I20251026 15:08:32.062675 23319 raft_consensus.cc:2243] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:32.062824 23319 raft_consensus.cc:2272] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:32.063987 23318 ts_tablet_manager.cc:1929] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:08:32.064061 23318 log.cc:1199] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/ef4d29a2974c405fb023ed2a7667604c
I20251026 15:08:32.064317 23318 ts_tablet_manager.cc:1950] T ef4d29a2974c405fb023ed2a7667604c P acab8578ae284981b41ec9aa59a0b645: Deleting consensus metadata
W20251026 15:08:32.066926 23155 connection.cc:537] server connection from 127.0.0.1:58682 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251026 15:08:32.067288 23317 ts_tablet_manager.cc:1916] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:32.068917 23317 ts_tablet_manager.cc:1929] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:08:32.068994 23317 log.cc:1199] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/ef4d29a2974c405fb023ed2a7667604c
I20251026 15:08:32.069270 23317 ts_tablet_manager.cc:1950] T ef4d29a2974c405fb023ed2a7667604c P 47e0e140c7734c9b980eb8527b45eb9a: Deleting consensus metadata
I20251026 15:08:32.066847 23319 ts_tablet_manager.cc:1916] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 15:08:32.069641 23019 connection.cc:441] server connection from 127.0.0.1:35030 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35030 (request call id 1) could send its response
I20251026 15:08:32.071170 23319 ts_tablet_manager.cc:1929] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:08:32.071249 23319 log.cc:1199] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-0/wal/wals/ef4d29a2974c405fb023ed2a7667604c
I20251026 15:08:32.071558 23319 ts_tablet_manager.cc:1950] T ef4d29a2974c405fb023ed2a7667604c P 1f3d80d6732a4f28980d053270c2d095: Deleting consensus metadata
W20251026 15:08:32.071951 22883 connection.cc:441] server connection from 127.0.0.1:56824 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56824 (request call id 1) could send its response
I20251026 15:08:32.075708 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:32.068923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.14.212.124:41613)
1026 15:08:32.068964 (+ 41us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:32.068966 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:32.068974 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:08:32.068999 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:32.069000 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:32.069012 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:32.069056 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:32.069849 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.070375 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:32.070955 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.071142 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:32.072788 (+ 1646us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:32.072792 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:32.072792 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:32.072799 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:32.074528 (+ 1729us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:32.074904 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:32.074905 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:32.074906 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:32.074935 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:32.075154 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:32.075156 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:32.075156 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:32.075277 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:32.075391 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:32.075525 (+ 134us) server_negotiation.cc:300] Negotiation successful
1026 15:08:32.075608 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20251026 15:08:32.080922 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:32.074687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.14.212.125:33059)
1026 15:08:32.074736 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:32.074739 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:32.074888 (+ 149us) server_negotiation.cc:408] Connection header received
1026 15:08:32.074996 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:32.074998 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:32.075010 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:32.075069 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:32.075847 (+ 778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.076482 (+ 635us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:32.077002 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:32.077188 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:32.078086 (+ 898us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:32.078090 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:32.078090 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:32.078096 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:32.080023 (+ 1927us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:32.080341 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:32.080342 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:32.080343 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:32.080365 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:32.080486 (+ 121us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:32.080486 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:32.080487 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:32.080589 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:32.080684 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:32.080740 (+ 56us) server_negotiation.cc:300] Negotiation successful
1026 15:08:32.080823 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20251026 15:08:33.057072 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:33.056767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.65:46777)
1026 15:08:33.056909 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:33.057006 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
W20251026 15:08:33.057267 22994 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:43071 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
I20251026 15:08:33.058321 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:33.058101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.67:53129)
1026 15:08:33.058217 (+ 116us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:33.058281 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
W20251026 15:08:33.058499 23266 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:43071 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
I20251026 15:08:33.058924 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:33.058669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.66:58591)
1026 15:08:33.058807 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:33.058882 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":55,"threads_started":1}
W20251026 15:08:33.059072 23130 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:43071 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
I20251026 15:08:33.559960 23329 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44a9eda249f24e0cb8b00854b910ffb9)
I20251026 15:08:33.559998 23330 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44a9eda249f24e0cb8b00854b910ffb9)
I20251026 15:08:33.560091 23330 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } }
I20251026 15:08:33.560096 23329 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } }
I20251026 15:08:33.560354 23329 leader_election.cc:290] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44a9eda249f24e0cb8b00854b910ffb9 (127.14.212.126:43071), 6ac476a3eab7475199c6e4d1d67593d3 (127.14.212.125:33059)
I20251026 15:08:33.560353 23330 leader_election.cc:290] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44a9eda249f24e0cb8b00854b910ffb9 (127.14.212.126:43071), 8c0b4a360db34b799231c882f33fbada (127.14.212.124:41613)
I20251026 15:08:33.560535 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c0b4a360db34b799231c882f33fbada" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6ac476a3eab7475199c6e4d1d67593d3" is_pre_election: true
I20251026 15:08:33.560631 22623 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c0b4a360db34b799231c882f33fbada in term 1.
I20251026 15:08:33.560654 22812 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ac476a3eab7475199c6e4d1d67593d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c0b4a360db34b799231c882f33fbada" is_pre_election: true
I20251026 15:08:33.560700 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:33.560461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37708)
1026 15:08:33.560595 (+ 134us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:33.560627 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":43,"threads_started":1}
I20251026 15:08:33.560726 22812 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ac476a3eab7475199c6e4d1d67593d3 in term 1.
I20251026 15:08:33.560738 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:33.560569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37710)
1026 15:08:33.560668 (+ 99us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:33.560701 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":41,"threads_started":1}
W20251026 15:08:33.560778 22598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
I20251026 15:08:33.560810 22789 leader_election.cc:304] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ac476a3eab7475199c6e4d1d67593d3, 8c0b4a360db34b799231c882f33fbada; no voters:
W20251026 15:08:33.560814 22786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
I20251026 15:08:33.560914 22598 leader_election.cc:304] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ac476a3eab7475199c6e4d1d67593d3, 8c0b4a360db34b799231c882f33fbada; no voters:
I20251026 15:08:33.560940 23329 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 15:08:33.560977 23329 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44a9eda249f24e0cb8b00854b910ffb9)
I20251026 15:08:33.561015 23329 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 1 FOLLOWER]: Advancing to term 2
I20251026 15:08:33.561005 23330 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 15:08:33.561048 23330 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44a9eda249f24e0cb8b00854b910ffb9)
I20251026 15:08:33.561076 23330 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 1 FOLLOWER]: Advancing to term 2
I20251026 15:08:33.561450 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:33.561359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37740)
1026 15:08:33.561399 (+ 40us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:33.561424 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20251026 15:08:33.561457 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:33.561324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37724)
1026 15:08:33.561405 (+ 81us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:33.561428 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":38,"threads_started":1}
W20251026 15:08:33.561518 22786 leader_election.cc:336] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 44a9eda249f24e0cb8b00854b910ffb9 (127.14.212.126:43071): Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
W20251026 15:08:33.561646 22598 leader_election.cc:336] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 44a9eda249f24e0cb8b00854b910ffb9 (127.14.212.126:43071): Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
I20251026 15:08:33.561895 23329 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } }
I20251026 15:08:33.561895 23330 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } }
I20251026 15:08:33.562049 23330 leader_election.cc:290] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [CANDIDATE]: Term 2 election: Requested vote from peers 44a9eda249f24e0cb8b00854b910ffb9 (127.14.212.126:43071), 8c0b4a360db34b799231c882f33fbada (127.14.212.124:41613)
I20251026 15:08:33.562062 23329 leader_election.cc:290] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [CANDIDATE]: Term 2 election: Requested vote from peers 44a9eda249f24e0cb8b00854b910ffb9 (127.14.212.126:43071), 6ac476a3eab7475199c6e4d1d67593d3 (127.14.212.125:33059)
I20251026 15:08:33.562278 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:33.562158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37744)
1026 15:08:33.562203 (+ 45us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:33.562230 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20251026 15:08:33.562287 22623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c0b4a360db34b799231c882f33fbada" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6ac476a3eab7475199c6e4d1d67593d3"
I20251026 15:08:33.562348 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:33.562267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37748)
1026 15:08:33.562300 (+ 33us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:33.562328 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20251026 15:08:33.562361 22623 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c0b4a360db34b799231c882f33fbada in current term 2: Already voted for candidate 6ac476a3eab7475199c6e4d1d67593d3 in this term.
I20251026 15:08:33.562420 22812 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ac476a3eab7475199c6e4d1d67593d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c0b4a360db34b799231c882f33fbada"
I20251026 15:08:33.562502 22812 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ac476a3eab7475199c6e4d1d67593d3 in current term 2: Already voted for candidate 8c0b4a360db34b799231c882f33fbada in this term.
I20251026 15:08:33.562837 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:33.562759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37754)
1026 15:08:33.562799 (+ 40us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:33.562820 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20251026 15:08:33.562950 22786 leader_election.cc:336] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 44a9eda249f24e0cb8b00854b910ffb9 (127.14.212.126:43071): Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
I20251026 15:08:33.563000 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:33.562819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37758)
1026 15:08:33.562859 (+ 40us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:33.562975 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20251026 15:08:33.563017 22786 leader_election.cc:304] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c0b4a360db34b799231c882f33fbada; no voters: 44a9eda249f24e0cb8b00854b910ffb9, 6ac476a3eab7475199c6e4d1d67593d3
W20251026 15:08:33.563112 22598 leader_election.cc:336] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 44a9eda249f24e0cb8b00854b910ffb9 (127.14.212.126:43071): Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
I20251026 15:08:33.563136 23329 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 15:08:33.563159 22598 leader_election.cc:304] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ac476a3eab7475199c6e4d1d67593d3; no voters: 44a9eda249f24e0cb8b00854b910ffb9, 8c0b4a360db34b799231c882f33fbada
I20251026 15:08:33.563225 23330 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 15:08:34.057823 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:34.057569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.65:58169)
1026 15:08:34.057708 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:34.057755 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20251026 15:08:34.059003 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:34.058754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.67:43289)
1026 15:08:34.058913 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:34.058952 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20251026 15:08:34.059504 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:34.059344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.66:37429)
1026 15:08:34.059436 (+ 92us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:34.059467 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":43,"threads_started":1}
I20251026 15:08:35.058722 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.058464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.65:38005)
1026 15:08:35.058607 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.058659 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20251026 15:08:35.059612 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.059402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.67:59209)
1026 15:08:35.059501 (+ 99us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.059558 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":47,"threads_started":1}
I20251026 15:08:35.060118 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.059914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.66:55027)
1026 15:08:35.060020 (+ 106us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.060048 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":51,"threads_started":1}
I20251026 15:08:35.201820 23342 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:08:35.201921 23342 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } }
I20251026 15:08:35.202107 23342 leader_election.cc:290] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 44a9eda249f24e0cb8b00854b910ffb9 (127.14.212.126:43071), 8c0b4a360db34b799231c882f33fbada (127.14.212.124:41613)
I20251026 15:08:35.202560 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.202331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37940)
1026 15:08:35.202473 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.202519 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":52,"threads_started":1}
I20251026 15:08:35.202652 22812 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ac476a3eab7475199c6e4d1d67593d3" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c0b4a360db34b799231c882f33fbada" is_pre_election: true
I20251026 15:08:35.202755 22812 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ac476a3eab7475199c6e4d1d67593d3 in term 2.
I20251026 15:08:35.203063 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.202950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37942)
1026 15:08:35.202984 (+ 34us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.203042 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20251026 15:08:35.203073 22598 leader_election.cc:304] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ac476a3eab7475199c6e4d1d67593d3, 8c0b4a360db34b799231c882f33fbada; no voters:
W20251026 15:08:35.203169 22598 leader_election.cc:336] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 44a9eda249f24e0cb8b00854b910ffb9 (127.14.212.126:43071): Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
I20251026 15:08:35.203174 23342 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 15:08:35.203229 23342 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:08:35.203264 23342 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 2 FOLLOWER]: Advancing to term 3
I20251026 15:08:35.205461 23342 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } }
I20251026 15:08:35.205605 23342 leader_election.cc:290] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [CANDIDATE]: Term 3 election: Requested vote from peers 44a9eda249f24e0cb8b00854b910ffb9 (127.14.212.126:43071), 8c0b4a360db34b799231c882f33fbada (127.14.212.124:41613)
I20251026 15:08:35.205850 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.205751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37958)
1026 15:08:35.205799 (+ 48us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.205831 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20251026 15:08:35.205929 22812 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ac476a3eab7475199c6e4d1d67593d3" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c0b4a360db34b799231c882f33fbada"
I20251026 15:08:35.206012 22812 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 2 FOLLOWER]: Advancing to term 3
I20251026 15:08:35.206735 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.206651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:37974)
1026 15:08:35.206692 (+ 41us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.206716 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20251026 15:08:35.206766 22812 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ac476a3eab7475199c6e4d1d67593d3 in term 3.
W20251026 15:08:35.206806 22598 leader_election.cc:336] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 44a9eda249f24e0cb8b00854b910ffb9 (127.14.212.126:43071): Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
I20251026 15:08:35.206936 22598 leader_election.cc:304] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ac476a3eab7475199c6e4d1d67593d3, 8c0b4a360db34b799231c882f33fbada; no voters: 44a9eda249f24e0cb8b00854b910ffb9
I20251026 15:08:35.207008 23342 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 15:08:35.207116 23342 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [term 3 LEADER]: Becoming Leader. State: Replica: 6ac476a3eab7475199c6e4d1d67593d3, State: Running, Role: LEADER
I20251026 15:08:35.207181 23342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } }
I20251026 15:08:35.207556 23345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ac476a3eab7475199c6e4d1d67593d3. Latest consensus state: current_term: 3 leader_uuid: "6ac476a3eab7475199c6e4d1d67593d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } } }
I20251026 15:08:35.207659 23345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:35.207836 23347 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:35.208189 23347 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:08:35.208280 23347 catalog_manager.cc:1269] Loaded cluster ID: 4b66c3f0fc7e41039c78c4f3971d7a0d
I20251026 15:08:35.208310 23347 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:08:35.208493 23347 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:35.208580 23347 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: Loaded TSK: 0
I20251026 15:08:35.208734 23347 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:08:35.230460 22614 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:08:35.231793 22614 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:08:35.232777 22812 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [term 3 FOLLOWER]: Refusing update from remote peer 6ac476a3eab7475199c6e4d1d67593d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20251026 15:08:35.232810 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.232692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:38004)
1026 15:08:35.232739 (+ 47us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.232774 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20251026 15:08:35.233031 23345 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20251026 15:08:35.233305 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.233174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.0.0.1:38010)
1026 15:08:35.233243 (+ 69us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.233276 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20251026 15:08:35.233404 22598 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 -> Peer 44a9eda249f24e0cb8b00854b910ffb9 (127.14.212.126:43071): Couldn't send request to peer 44a9eda249f24e0cb8b00854b910ffb9. Status: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 15:08:35.233677 23349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ac476a3eab7475199c6e4d1d67593d3. Latest consensus state: current_term: 3 leader_uuid: "6ac476a3eab7475199c6e4d1d67593d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } } }
I20251026 15:08:35.233769 23349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:08:35.234068 23348 mvcc.cc:204] Tried to move back new op lower bound from 7215068427191578624 to 7215068427089186816. Current Snapshot: MvccSnapshot[applied={T|T < 7215068427191578624}]
I20251026 15:08:35.234169 23342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6ac476a3eab7475199c6e4d1d67593d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } } }
I20251026 15:08:35.234246 23342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:35.234311 23349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6ac476a3eab7475199c6e4d1d67593d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44a9eda249f24e0cb8b00854b910ffb9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43071 } } peers { permanent_uuid: "6ac476a3eab7475199c6e4d1d67593d3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 33059 } attrs { promote: false } } peers { permanent_uuid: "8c0b4a360db34b799231c882f33fbada" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41613 } attrs { promote: false } } }
I20251026 15:08:35.234447 23349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0b4a360db34b799231c882f33fbada [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:08:35.238495 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.235834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46657 (local address 127.0.0.1:35046)
1026 15:08:35.236011 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.236021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:35.236086 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:35.236423 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:35.236424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:35.236427 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:35.236609 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:35.236611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.237412 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:35.237413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:35.238248 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:35.238250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.238320 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:35.238325 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:08:35.238357 (+ 32us) client_negotiation.cc:240] Negotiation successful
1026 15:08:35.238416 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":76,"threads_started":1}
I20251026 15:08:35.238809 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.235921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:43321 (local address 127.0.0.1:56836)
1026 15:08:35.236163 (+ 242us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.236173 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:35.236212 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:35.236589 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:35.236590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:35.236592 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:35.236768 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:35.236769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.237687 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:35.237688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:35.238533 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:35.238535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.238677 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:35.238680 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:08:35.238709 (+ 29us) client_negotiation.cc:240] Negotiation successful
1026 15:08:35.238746 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":81,"threads_started":1}
I20251026 15:08:35.238828 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.235799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:46625 (local address 127.0.0.1:58712)
1026 15:08:35.235832 (+ 33us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.235843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:35.235895 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:35.236756 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:35.236757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:35.236758 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:35.236955 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:35.236956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.237800 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:35.237801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:35.238591 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:35.238594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.238662 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:35.238666 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:35.238701 (+ 35us) client_negotiation.cc:240] Negotiation successful
1026 15:08:35.238747 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20251026 15:08:35.239387 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.235972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58712 (local address 127.14.212.67:46625)
1026 15:08:35.236285 (+ 313us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:35.236287 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:35.236304 (+ 17us) server_negotiation.cc:408] Connection header received
1026 15:08:35.236331 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:35.236332 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:35.236344 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:35.236397 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:35.237016 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.237725 (+ 709us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:35.238855 (+ 1130us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.239282 (+ 427us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:35.239288 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:35.239308 (+ 20us) server_negotiation.cc:300] Negotiation successful
1026 15:08:35.239338 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":70,"threads_started":1}
I20251026 15:08:35.239615 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.235998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.14.212.65:43321)
1026 15:08:35.236476 (+ 478us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:35.236478 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:35.236490 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:08:35.236510 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:35.236510 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:35.236519 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:35.236573 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:35.236856 (+ 283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.237608 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:35.239126 (+ 1518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.239487 (+ 361us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:35.239496 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:35.239522 (+ 26us) server_negotiation.cc:300] Negotiation successful
1026 15:08:35.239564 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":60,"threads_started":1}
I20251026 15:08:35.239615 23200 tablet_service.cc:1505] Processing CreateTablet for tablet 90c447bd3b4343db8d550701ea63dd89 (DEFAULT_TABLE table=test-table [id=a0433c7d868a4fa0bcba22014ed5e551]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:35.239743 23200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c447bd3b4343db8d550701ea63dd89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:35.240792 23360 tablet_bootstrap.cc:492] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645: Bootstrap starting.
I20251026 15:08:35.240823 22928 tablet_service.cc:1505] Processing CreateTablet for tablet 90c447bd3b4343db8d550701ea63dd89 (DEFAULT_TABLE table=test-table [id=a0433c7d868a4fa0bcba22014ed5e551]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:35.240947 22928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c447bd3b4343db8d550701ea63dd89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:35.241199 23360 tablet_bootstrap.cc:654] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:35.241750 23360 tablet_bootstrap.cc:492] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645: No bootstrap required, opened a new log
I20251026 15:08:35.241784 23360 ts_tablet_manager.cc:1403] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:35.241892 23362 tablet_bootstrap.cc:492] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095: Bootstrap starting.
I20251026 15:08:35.241878 23360 raft_consensus.cc:359] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:35.241940 23360 raft_consensus.cc:385] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:35.241966 23360 raft_consensus.cc:740] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acab8578ae284981b41ec9aa59a0b645, State: Initialized, Role: FOLLOWER
I20251026 15:08:35.242063 23360 consensus_queue.cc:260] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:35.242247 23360 ts_tablet_manager.cc:1434] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:35.242340 23362 tablet_bootstrap.cc:654] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:35.242609 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.242489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.67:40633)
1026 15:08:35.242537 (+ 48us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.242575 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20251026 15:08:35.243062 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.236095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35046 (local address 127.14.212.66:46657)
1026 15:08:35.236264 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:35.236266 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:35.236283 (+ 17us) server_negotiation.cc:408] Connection header received
1026 15:08:35.236306 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:35.236307 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:35.236317 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:35.236395 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:35.236683 (+ 288us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.237319 (+ 636us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:35.238462 (+ 1143us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.242926 (+ 4464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:35.242934 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:35.242955 (+ 21us) server_negotiation.cc:300] Negotiation successful
1026 15:08:35.242986 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20251026 15:08:35.243132 23268 heartbeater.cc:499] Master 127.14.212.125:33059 was elected leader, sending a full tablet report...
I20251026 15:08:35.243265 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.243160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.67:48079)
1026 15:08:35.243198 (+ 38us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.243240 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20251026 15:08:35.243499 23064 tablet_service.cc:1505] Processing CreateTablet for tablet 90c447bd3b4343db8d550701ea63dd89 (DEFAULT_TABLE table=test-table [id=a0433c7d868a4fa0bcba22014ed5e551]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:35.243645 23064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c447bd3b4343db8d550701ea63dd89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:35.244650 23365 tablet_bootstrap.cc:492] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a: Bootstrap starting.
I20251026 15:08:35.244930 23362 tablet_bootstrap.cc:492] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095: No bootstrap required, opened a new log
I20251026 15:08:35.244987 23362 ts_tablet_manager.cc:1403] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251026 15:08:35.245065 23365 tablet_bootstrap.cc:654] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:35.245167 23362 raft_consensus.cc:359] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:35.245235 23362 raft_consensus.cc:385] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:35.245250 23362 raft_consensus.cc:740] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f3d80d6732a4f28980d053270c2d095, State: Initialized, Role: FOLLOWER
I20251026 15:08:35.245327 23362 consensus_queue.cc:260] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:35.245498 23362 ts_tablet_manager.cc:1434] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:35.245755 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.245646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.65:43937)
1026 15:08:35.245696 (+ 50us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.245727 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20251026 15:08:35.245731 23365 tablet_bootstrap.cc:492] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a: No bootstrap required, opened a new log
I20251026 15:08:35.245945 22996 heartbeater.cc:499] Master 127.14.212.125:33059 was elected leader, sending a full tablet report...
I20251026 15:08:35.246029 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.245950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.65:58195)
1026 15:08:35.245981 (+ 31us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.246004 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20251026 15:08:35.246055 23365 ts_tablet_manager.cc:1403] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:35.246213 23365 raft_consensus.cc:359] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:35.246320 23365 raft_consensus.cc:385] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:35.246425 23365 raft_consensus.cc:740] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e0e140c7734c9b980eb8527b45eb9a, State: Initialized, Role: FOLLOWER
I20251026 15:08:35.246515 23365 consensus_queue.cc:260] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:35.246668 23365 ts_tablet_manager.cc:1434] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:35.246942 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.246825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.66:57697)
1026 15:08:35.246875 (+ 50us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.246910 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20251026 15:08:35.247218 23132 heartbeater.cc:499] Master 127.14.212.125:33059 was elected leader, sending a full tablet report...
I20251026 15:08:35.247264 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.247178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.66:33303)
1026 15:08:35.247212 (+ 34us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.247238 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20251026 15:08:35.382196 22997 tablet.cc:2378] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:08:35.394802 23367 raft_consensus.cc:493] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:08:35.394919 23367 raft_consensus.cc:515] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:35.395259 23367 leader_election.cc:290] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acab8578ae284981b41ec9aa59a0b645 (127.14.212.67:46625), 47e0e140c7734c9b980eb8527b45eb9a (127.14.212.66:46657)
I20251026 15:08:35.397583 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.395467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:46625 (local address 127.14.212.65:60921)
1026 15:08:35.395656 (+ 189us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.395664 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:35.395727 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:35.395928 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:35.395929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:35.395930 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:35.396143 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:35.396146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.396787 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:35.396788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:35.397397 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:35.397399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.397461 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:35.397464 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:08:35.397495 (+ 31us) client_negotiation.cc:240] Negotiation successful
1026 15:08:35.397522 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":14,"thread_start_us":114,"threads_started":1}
I20251026 15:08:35.397848 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.395467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46657 (local address 127.14.212.65:57241)
1026 15:08:35.395540 (+ 73us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.395554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:35.395610 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:35.395876 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:35.395877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:35.395880 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:35.396163 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:35.396165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.396963 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:35.396964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:35.397615 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:35.397617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.397693 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:35.397697 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:35.397729 (+ 32us) client_negotiation.cc:240] Negotiation successful
1026 15:08:35.397783 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20251026 15:08:35.398183 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.395595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:60921 (local address 127.14.212.67:46625)
1026 15:08:35.395642 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:35.395644 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:35.395733 (+ 89us) server_negotiation.cc:408] Connection header received
1026 15:08:35.395791 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:35.395792 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:35.395802 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:35.395876 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:35.396224 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.396711 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:35.397514 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.398046 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:35.398056 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:35.398083 (+ 27us) server_negotiation.cc:300] Negotiation successful
1026 15:08:35.398122 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251026 15:08:35.398432 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.395704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:57241 (local address 127.14.212.66:46657)
1026 15:08:35.395752 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:35.395754 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:35.395762 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:08:35.395790 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:35.395791 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:35.395801 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:35.395855 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:35.396256 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.396788 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:35.397794 (+ 1006us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.398250 (+ 456us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:35.398258 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:35.398286 (+ 28us) server_negotiation.cc:300] Negotiation successful
1026 15:08:35.398333 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251026 15:08:35.398487 23220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c447bd3b4343db8d550701ea63dd89" candidate_uuid: "1f3d80d6732a4f28980d053270c2d095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acab8578ae284981b41ec9aa59a0b645" is_pre_election: true
I20251026 15:08:35.398592 23220 raft_consensus.cc:2468] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f3d80d6732a4f28980d053270c2d095 in term 0.
I20251026 15:08:35.398651 23084 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c447bd3b4343db8d550701ea63dd89" candidate_uuid: "1f3d80d6732a4f28980d053270c2d095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e0e140c7734c9b980eb8527b45eb9a" is_pre_election: true
I20251026 15:08:35.398732 23084 raft_consensus.cc:2468] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f3d80d6732a4f28980d053270c2d095 in term 0.
I20251026 15:08:35.398762 22883 leader_election.cc:304] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f3d80d6732a4f28980d053270c2d095, acab8578ae284981b41ec9aa59a0b645; no voters:
I20251026 15:08:35.398895 23367 raft_consensus.cc:2804] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:08:35.398963 23367 raft_consensus.cc:493] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:08:35.398989 23367 raft_consensus.cc:3060] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:35.399612 23367 raft_consensus.cc:515] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:35.399732 23367 leader_election.cc:290] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [CANDIDATE]: Term 1 election: Requested vote from peers acab8578ae284981b41ec9aa59a0b645 (127.14.212.67:46625), 47e0e140c7734c9b980eb8527b45eb9a (127.14.212.66:46657)
I20251026 15:08:35.399899 23220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c447bd3b4343db8d550701ea63dd89" candidate_uuid: "1f3d80d6732a4f28980d053270c2d095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acab8578ae284981b41ec9aa59a0b645"
I20251026 15:08:35.399926 23084 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c447bd3b4343db8d550701ea63dd89" candidate_uuid: "1f3d80d6732a4f28980d053270c2d095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e0e140c7734c9b980eb8527b45eb9a"
I20251026 15:08:35.399968 23220 raft_consensus.cc:3060] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:35.399981 23084 raft_consensus.cc:3060] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:35.400552 23084 raft_consensus.cc:2468] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f3d80d6732a4f28980d053270c2d095 in term 1.
I20251026 15:08:35.400655 23220 raft_consensus.cc:2468] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f3d80d6732a4f28980d053270c2d095 in term 1.
I20251026 15:08:35.400751 22881 leader_election.cc:304] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f3d80d6732a4f28980d053270c2d095, 47e0e140c7734c9b980eb8527b45eb9a; no voters:
I20251026 15:08:35.400856 23367 raft_consensus.cc:2804] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:35.400990 23367 raft_consensus.cc:697] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [term 1 LEADER]: Becoming Leader. State: Replica: 1f3d80d6732a4f28980d053270c2d095, State: Running, Role: LEADER
I20251026 15:08:35.401042 23367 consensus_queue.cc:237] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } } peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } }
I20251026 15:08:35.401443 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.401357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.65:54543)
1026 15:08:35.401389 (+ 32us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.401420 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20251026 15:08:35.401695 22614 catalog_manager.cc:5649] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f3d80d6732a4f28980d053270c2d095 (127.14.212.65). New cstate: current_term: 1 leader_uuid: "1f3d80d6732a4f28980d053270c2d095" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f3d80d6732a4f28980d053270c2d095" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43321 } health_report { overall_health: HEALTHY } } }
W20251026 15:08:35.404660 23269 tablet.cc:2378] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:08:35.422363 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.420344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.14.212.65:43321)
1026 15:08:35.420403 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:35.420405 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:35.420414 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:08:35.420453 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:35.420454 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:35.420472 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:35.420529 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:08:35.420734 (+ 205us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.421226 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:35.421821 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:35.421938 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:35.421967 (+ 29us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:08:35.422059 (+ 92us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:08:35.422137 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:35.422244 (+ 107us) server_negotiation.cc:300] Negotiation successful
1026 15:08:35.422277 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251026 15:08:35.423538 23220 raft_consensus.cc:1275] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645 [term 1 FOLLOWER]: Refusing update from remote peer 1f3d80d6732a4f28980d053270c2d095: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:35.423732 23084 raft_consensus.cc:1275] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a [term 1 FOLLOWER]: Refusing update from remote peer 1f3d80d6732a4f28980d053270c2d095: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:35.423825 23367 consensus_queue.cc:1048] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acab8578ae284981b41ec9aa59a0b645" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:35.424036 23367 consensus_queue.cc:1048] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e0e140c7734c9b980eb8527b45eb9a" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:35.424710 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.424588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.67:40953)
1026 15:08:35.424634 (+ 46us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.424669 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20251026 15:08:35.424731 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.424637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.66:60949)
1026 15:08:35.424676 (+ 39us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.424700 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20251026 15:08:35.425966 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.425721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.65:58413)
1026 15:08:35.425901 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.425936 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20251026 15:08:35.426493 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.426345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.66:51687)
1026 15:08:35.426434 (+ 89us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.426459 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20251026 15:08:35.428393 22613 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58688:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:35.428480 22613 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58688:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:35.428558 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.428455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.67:37601)
1026 15:08:35.428494 (+ 39us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.428519 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20251026 15:08:35.428920 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.428823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43071 (local address 127.14.212.65:49627)
1026 15:08:35.428862 (+ 39us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.428890 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20251026 15:08:35.429642 23374 mvcc.cc:204] Tried to move back new op lower bound from 7215068427972456448 to 7215068427883212800. Current Snapshot: MvccSnapshot[applied={T|T < 7215068427972456448 or (T in {7215068427972456448})}]
I20251026 15:08:35.430056 22613 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 6ac476a3eab7475199c6e4d1d67593d3: Sending DeleteTablet for 3 replicas of tablet 90c447bd3b4343db8d550701ea63dd89
I20251026 15:08:35.430320 23200 tablet_service.cc:1552] Processing DeleteTablet for tablet 90c447bd3b4343db8d550701ea63dd89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:35 UTC) from {username='oryx', principal='oryx/127.14.212.125@KRBTEST.COM'} at 127.0.0.1:58712
I20251026 15:08:35.430377 23064 tablet_service.cc:1552] Processing DeleteTablet for tablet 90c447bd3b4343db8d550701ea63dd89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:35 UTC) from {username='oryx', principal='oryx/127.14.212.125@KRBTEST.COM'} at 127.0.0.1:35046
I20251026 15:08:35.430562 22928 tablet_service.cc:1552] Processing DeleteTablet for tablet 90c447bd3b4343db8d550701ea63dd89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:35 UTC) from {username='oryx', principal='oryx/127.14.212.125@KRBTEST.COM'} at 127.0.0.1:56836
I20251026 15:08:35.430727 23382 tablet_replica.cc:333] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645: stopping tablet replica
I20251026 15:08:35.430756 23384 tablet_replica.cc:333] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095: stopping tablet replica
I20251026 15:08:35.430832 23384 raft_consensus.cc:2243] T 90c447bd3b4343db8d550701ea63dd89 P 1f3d80d6732a4f28980d053270c2d095 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:08:35.430825 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 22865
I20251026 15:08:35.432723 23383 tablet_replica.cc:333] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a: stopping tablet replica
I20251026 15:08:35.432787 23383 raft_consensus.cc:2243] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:35.432847 23383 raft_consensus.cc:2272] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:35.432941 23382 raft_consensus.cc:2243] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:35.433013 23382 raft_consensus.cc:2272] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:35.433269 23382 ts_tablet_manager.cc:1916] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:35.434793 23382 ts_tablet_manager.cc:1929] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:08:35.434854 23382 log.cc:1199] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/90c447bd3b4343db8d550701ea63dd89
I20251026 15:08:35.435118 23382 ts_tablet_manager.cc:1950] T 90c447bd3b4343db8d550701ea63dd89 P acab8578ae284981b41ec9aa59a0b645: Deleting consensus metadata
I20251026 15:08:35.435437 22600 catalog_manager.cc:4985] TS acab8578ae284981b41ec9aa59a0b645 (127.14.212.67:46625): tablet 90c447bd3b4343db8d550701ea63dd89 (table test-table [id=a0433c7d868a4fa0bcba22014ed5e551]) successfully deleted
I20251026 15:08:35.435667 23383 ts_tablet_manager.cc:1916] T 90c447bd3b4343db8d550701ea63dd89 P 47e0e140c7734c9b980eb8527b45eb9a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 15:08:35.436228 23343 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:35.435551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:43321 (local address 127.0.0.1:56846)
1026 15:08:35.435791 (+ 240us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.435804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:35.435847 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:35.436175 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:43321: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":222}
W20251026 15:08:35.436332 22599 catalog_manager.cc:4712] TS 1f3d80d6732a4f28980d053270c2d095 (127.14.212.65:43321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90c447bd3b4343db8d550701ea63dd89: Network error: Client connection negotiation failed: client connection to 127.14.212.65:43321: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251026 15:08:35.436497 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 23001
I20251026 15:08:35.441720 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:35.441606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46657 (local address 127.0.0.1:35060)
1026 15:08:35.441659 (+ 53us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:35.441684 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.66:46657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20251026 15:08:35.441790 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 23137
I20251026 15:08:35.447753 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 22479
I20251026 15:08:35.452575 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 22670
2025-10-26T15:08:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8465 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:08:35 dist-test-slave-8pgz krb5kdc[23387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:08:35 dist-test-slave-8pgz krb5kdc[23387](info): set up 2 sockets
Oct 26 15:08:35 dist-test-slave-8pgz krb5kdc[23387](info): commencing operation
krb5kdc: starting...
W20251026 15:08:37.460538 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.985s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:08:37 dist-test-slave-8pgz krb5kdc[23387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:08:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
I20251026 15:08:37.594702 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33841
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:33841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:37.676450 23403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:37.676637 23403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:37.676673 23403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:37.678145 23403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:37.678193 23403 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251026 15:08:37.678215 23403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:37.678236 23403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:37.678254 23403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:37.678275 23403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:08:37.679884 23403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:33841
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:37.680145 23403 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:37.680328 23403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:08:37.683208 23403 server_base.cc:1047] running on GCE node
W20251026 15:08:37.683157 23408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:37.683207 23411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:37.683177 23409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:37.683708 23403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:37.683933 23403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:37.685070 23403 hybrid_clock.cc:648] HybridClock initialized: now 1761491317685050 us; error 34 us; skew 500 ppm
Oct 26 15:08:37 dist-test-slave-8pgz krb5kdc[23387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491317, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:37.687491 23403 init.cc:377] Logged in from keytab as oryx/127.14.212.126@KRBTEST.COM (short username oryx)
I20251026 15:08:37.688006 23403 webserver.cc:492] Webserver started at http://127.14.212.126:33685/ using document root <none> and password file <none>
I20251026 15:08:37.688194 23403 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:37.688234 23403 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:37.688346 23403 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:37.689389 23403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "3ec2c8f4473247ceaf108e8dc22c800a"
format_stamp: "Formatted at 2025-10-26 15:08:37 on dist-test-slave-8pgz"
server_key: "e69ba94144c054569d825d05b448fab5"
server_key_iv: "b0c5c0176fe118475a999a2ddea282e0"
server_key_version: "encryptionkey@0"
I20251026 15:08:37.689690 23403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "3ec2c8f4473247ceaf108e8dc22c800a"
format_stamp: "Formatted at 2025-10-26 15:08:37 on dist-test-slave-8pgz"
server_key: "e69ba94144c054569d825d05b448fab5"
server_key_iv: "b0c5c0176fe118475a999a2ddea282e0"
server_key_version: "encryptionkey@0"
I20251026 15:08:37.691166 23403 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20251026 15:08:37.691931 23418 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:37.692166 23403 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:37.692251 23403 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "3ec2c8f4473247ceaf108e8dc22c800a"
format_stamp: "Formatted at 2025-10-26 15:08:37 on dist-test-slave-8pgz"
server_key: "e69ba94144c054569d825d05b448fab5"
server_key_iv: "b0c5c0176fe118475a999a2ddea282e0"
server_key_version: "encryptionkey@0"
I20251026 15:08:37.692313 23403 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:37.712321 23403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:37.714270 23403 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:37.714434 23403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:37.718874 23403 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:33841
I20251026 15:08:37.718910 23480 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:33841 every 8 connection(s)
I20251026 15:08:37.719250 23403 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:37.720103 23481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:37.722239 23481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a: Bootstrap starting.
I20251026 15:08:37.722976 23481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:37.723229 23481 log.cc:826] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:37.723905 23481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a: No bootstrap required, opened a new log
I20251026 15:08:37.725095 23481 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec2c8f4473247ceaf108e8dc22c800a" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33841 } }
I20251026 15:08:37.725234 23481 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:37.725255 23481 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec2c8f4473247ceaf108e8dc22c800a, State: Initialized, Role: FOLLOWER
I20251026 15:08:37.725365 23481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec2c8f4473247ceaf108e8dc22c800a" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33841 } }
I20251026 15:08:37.725445 23481 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:37.725488 23481 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:37.725522 23481 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:37.726020 23481 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec2c8f4473247ceaf108e8dc22c800a" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33841 } }
I20251026 15:08:37.726118 23481 leader_election.cc:304] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ec2c8f4473247ceaf108e8dc22c800a; no voters:
I20251026 15:08:37.726239 23481 leader_election.cc:290] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:08:37.726325 23484 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:37.726490 23484 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [term 1 LEADER]: Becoming Leader. State: Replica: 3ec2c8f4473247ceaf108e8dc22c800a, State: Running, Role: LEADER
I20251026 15:08:37.726501 23481 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:37.726642 23484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec2c8f4473247ceaf108e8dc22c800a" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33841 } }
I20251026 15:08:37.726998 23485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ec2c8f4473247ceaf108e8dc22c800a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec2c8f4473247ceaf108e8dc22c800a" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33841 } } }
I20251026 15:08:37.727038 23486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ec2c8f4473247ceaf108e8dc22c800a. Latest consensus state: current_term: 1 leader_uuid: "3ec2c8f4473247ceaf108e8dc22c800a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec2c8f4473247ceaf108e8dc22c800a" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33841 } } }
I20251026 15:08:37.727133 23485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:37.727144 23486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:37.727880 23489 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:37.728313 23489 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:08:37.729151 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 23403
I20251026 15:08:37.729245 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:08:37.729421 15185 external_mini_cluster.cc:1468] Setting key ccb1836b6eea7e7cb7a8772f9e62d09f
I20251026 15:08:37.729796 23489 catalog_manager.cc:1357] Generated new cluster ID: 39ff4890dc974721a1e3e5fa056afc6d
I20251026 15:08:37.729844 23489 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 26 15:08:37 dist-test-slave-8pgz krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
I20251026 15:08:37.737169 23489 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:08:37.738248 23489 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:37.739620 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:37.729889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.14.212.126:33841)
1026 15:08:37.730051 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:37.730063 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:37.730184 (+ 121us) server_negotiation.cc:408] Connection header received
1026 15:08:37.730329 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:37.730334 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:37.730379 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:37.730518 (+ 139us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:37.731522 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:37.732680 (+ 1158us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:37.733644 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:37.733876 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:37.736180 (+ 2304us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:37.736187 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:37.736188 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:37.736210 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:37.738271 (+ 2061us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:37.738650 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:37.738653 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:37.738654 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:37.738716 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:37.738864 (+ 148us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:37.738865 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:37.738865 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:37.739053 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:37.739182 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:37.739302 (+ 120us) server_negotiation.cc:300] Negotiation successful
1026 15:08:37.739448 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20251026 15:08:37.749226 23489 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a: Generated new TSK 0
I20251026 15:08:37.749450 23489 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
I20251026 15:08:37.799440 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33841
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:37.880337 23509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:37.880525 23509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:37.880564 23509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:37.882095 23509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:37.882148 23509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:37.882177 23509 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 15:08:37.882206 23509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:08:37.883841 23509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.212.126:33841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:37.884089 23509 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:37.884269 23509 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:37.886982 23515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:37.886993 23514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:37.887223 23509 server_base.cc:1047] running on GCE node
W20251026 15:08:37.887318 23517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:37.887494 23509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:37.887706 23509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:37.888839 23509 hybrid_clock.cc:648] HybridClock initialized: now 1761491317888818 us; error 39 us; skew 500 ppm
Oct 26 15:08:37 dist-test-slave-8pgz krb5kdc[23387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491317, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:37.891151 23509 init.cc:377] Logged in from keytab as oryx/127.14.212.65@KRBTEST.COM (short username oryx)
I20251026 15:08:37.891649 23509 webserver.cc:492] Webserver started at http://127.14.212.65:35557/ using document root <none> and password file <none>
I20251026 15:08:37.891835 23509 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:37.891876 23509 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:37.891990 23509 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:37.892933 23509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "e9473e4ac53945b2ae886fd889a8cef0"
format_stamp: "Formatted at 2025-10-26 15:08:37 on dist-test-slave-8pgz"
server_key: "15b4d75cfbafd60b9ea3ede86b66c85a"
server_key_iv: "c8757f756b97f832d98d5a89e9302a6f"
server_key_version: "encryptionkey@0"
I20251026 15:08:37.893252 23509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "e9473e4ac53945b2ae886fd889a8cef0"
format_stamp: "Formatted at 2025-10-26 15:08:37 on dist-test-slave-8pgz"
server_key: "15b4d75cfbafd60b9ea3ede86b66c85a"
server_key_iv: "c8757f756b97f832d98d5a89e9302a6f"
server_key_version: "encryptionkey@0"
I20251026 15:08:37.894552 23509 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:08:37.895282 23524 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:37.895466 23509 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:37.895550 23509 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "e9473e4ac53945b2ae886fd889a8cef0"
format_stamp: "Formatted at 2025-10-26 15:08:37 on dist-test-slave-8pgz"
server_key: "15b4d75cfbafd60b9ea3ede86b66c85a"
server_key_iv: "c8757f756b97f832d98d5a89e9302a6f"
server_key_version: "encryptionkey@0"
I20251026 15:08:37.895623 23509 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:37.918068 23509 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:37.920465 23509 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:37.920631 23509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:37.921208 23509 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:37.921280 23509 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:37.921324 23509 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:37.921360 23509 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 15:08:37 dist-test-slave-8pgz krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491317, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
I20251026 15:08:37.928979 23509 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:41571
I20251026 15:08:37.929035 23639 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:41571 every 8 connection(s)
I20251026 15:08:37.929421 23509 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:08:37.931459 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:37.921706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:53571 (local address 127.14.212.126:33841)
1026 15:08:37.921759 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:37.921761 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:37.921961 (+ 200us) server_negotiation.cc:408] Connection header received
1026 15:08:37.922551 (+ 590us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:37.922553 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:37.922571 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:37.922628 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:37.923731 (+ 1103us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:37.924601 (+ 870us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:37.925518 (+ 917us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:37.925756 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:37.928592 (+ 2836us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:37.928597 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:37.928598 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:37.928606 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:37.930293 (+ 1687us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:37.930714 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:37.930715 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:37.930716 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:37.930736 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:37.930911 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:37.930912 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:37.930912 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:37.930997 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:37.931042 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:37.931303 (+ 261us) server_negotiation.cc:300] Negotiation successful
1026 15:08:37.931396 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251026 15:08:37.931524 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:37.921687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33841 (local address 127.14.212.65:53571)
1026 15:08:37.921841 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:37.921868 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:37.922429 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:37.922690 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:37.922695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:37.923140 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:37.923595 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:37.923600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:37.924710 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:37.924711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:37.925415 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:37.925417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:37.925579 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:37.926281 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:37.926307 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:37.928467 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:37.930389 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:37.930439 (+ 50us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:37.930443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:37.930651 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:37.930802 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:37.930803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:37.930803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:37.930863 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:37.931077 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:37.931080 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:37.931194 (+ 114us) client_negotiation.cc:769] Sending connection context
1026 15:08:37.931276 (+ 82us) client_negotiation.cc:240] Negotiation successful
1026 15:08:37.931386 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":103,"threads_started":1}
I20251026 15:08:37.932018 23640 heartbeater.cc:344] Connected to a master server at 127.14.212.126:33841
I20251026 15:08:37.932096 23640 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:37.932301 23640 heartbeater.cc:507] Master 127.14.212.126:33841 requested a full tablet report, sending...
I20251026 15:08:37.932753 23435 ts_manager.cc:194] Registered new tserver with Master: e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571)
I20251026 15:08:37.933490 23435 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:53571
I20251026 15:08:37.934443 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 23509
I20251026 15:08:37.934551 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:08:37.934705 15185 external_mini_cluster.cc:1468] Setting key 3f9efd76d185fc21b489c7c2414ce270
I20251026 15:08:37.938722 23646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:37.932996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33841 (local address 127.14.212.65:34015)
1026 15:08:37.933151 (+ 155us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:37.933160 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:37.933219 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:37.933345 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:37.933346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:37.933492 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:37.933699 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:37.933700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:37.934321 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:37.934322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:37.934752 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:37.934754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:37.934838 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:37.935490 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:37.935494 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:37.935823 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:37.937798 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:37.937799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:37.937799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:37.937956 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:37.938103 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:37.938104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:37.938104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:37.938132 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:37.938323 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:37.938324 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:37.938367 (+ 43us) client_negotiation.cc:769] Sending connection context
1026 15:08:37.938461 (+ 94us) client_negotiation.cc:240] Negotiation successful
1026 15:08:37.938583 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20251026 15:08:37.938856 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:37.933047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:34015 (local address 127.14.212.126:33841)
1026 15:08:37.933087 (+ 40us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:37.933088 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:37.933215 (+ 127us) server_negotiation.cc:408] Connection header received
1026 15:08:37.933276 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:37.933277 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:37.933288 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:37.933342 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:37.933769 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:37.934234 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:37.934851 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:37.934993 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:37.935924 (+ 931us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:37.935928 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:37.935928 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:37.935934 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:37.937716 (+ 1782us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:37.938020 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:37.938021 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:37.938021 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:37.938044 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:37.938185 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:37.938186 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:37.938186 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:37.938270 (+ 84us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:37.938330 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:37.938529 (+ 199us) server_negotiation.cc:300] Negotiation successful
1026 15:08:37.938689 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
I20251026 15:08:37.985289 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33841
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:38.065791 23651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:38.065968 23651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:38.065994 23651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:38.067523 23651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:38.067574 23651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:38.067595 23651 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 15:08:38.067611 23651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:08:38.069105 23651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.212.126:33841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:38.069324 23651 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:38.069483 23651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:38.072144 23657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:38.072198 23651 server_base.cc:1047] running on GCE node
W20251026 15:08:38.072149 23656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:38.072149 23659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:38.072609 23651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:38.072839 23651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:38.073987 23651 hybrid_clock.cc:648] HybridClock initialized: now 1761491318073968 us; error 36 us; skew 500 ppm
Oct 26 15:08:38 dist-test-slave-8pgz krb5kdc[23387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491318, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:38.076443 23651 init.cc:377] Logged in from keytab as oryx/127.14.212.66@KRBTEST.COM (short username oryx)
I20251026 15:08:38.076967 23651 webserver.cc:492] Webserver started at http://127.14.212.66:42189/ using document root <none> and password file <none>
I20251026 15:08:38.077159 23651 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:38.077199 23651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:38.077311 23651 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:38.078279 23651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "716583253ac44e71b7c36c57e3720a0c"
format_stamp: "Formatted at 2025-10-26 15:08:38 on dist-test-slave-8pgz"
server_key: "4b16d4ca3a9a4717df392f98edc355ef"
server_key_iv: "44637e9c6836acb5d96fe1c77ba60527"
server_key_version: "encryptionkey@0"
I20251026 15:08:38.078619 23651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "716583253ac44e71b7c36c57e3720a0c"
format_stamp: "Formatted at 2025-10-26 15:08:38 on dist-test-slave-8pgz"
server_key: "4b16d4ca3a9a4717df392f98edc355ef"
server_key_iv: "44637e9c6836acb5d96fe1c77ba60527"
server_key_version: "encryptionkey@0"
I20251026 15:08:38.079859 23651 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:38.080536 23666 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:38.080714 23651 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:08:38.080785 23651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "716583253ac44e71b7c36c57e3720a0c"
format_stamp: "Formatted at 2025-10-26 15:08:38 on dist-test-slave-8pgz"
server_key: "4b16d4ca3a9a4717df392f98edc355ef"
server_key_iv: "44637e9c6836acb5d96fe1c77ba60527"
server_key_version: "encryptionkey@0"
I20251026 15:08:38.080845 23651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:38.092535 23651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:38.094278 23651 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:38.094431 23651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:38.094944 23651 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:38.094995 23651 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:38.095023 23651 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:38.095041 23651 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 15:08:38 dist-test-slave-8pgz krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491318, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
I20251026 15:08:38.101938 23651 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:35369
I20251026 15:08:38.101999 23781 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:35369 every 8 connection(s)
I20251026 15:08:38.102278 23651 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:08:38.105075 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.095406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:46783 (local address 127.14.212.126:33841)
1026 15:08:38.095485 (+ 79us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.095487 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.095715 (+ 228us) server_negotiation.cc:408] Connection header received
1026 15:08:38.096530 (+ 815us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.096531 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.096544 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.096618 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:38.097787 (+ 1169us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.098646 (+ 859us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.099632 (+ 986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.099840 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.102362 (+ 2522us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:38.102367 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:38.102367 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:38.102376 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:38.103828 (+ 1452us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.104232 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.104233 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.104233 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.104260 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.104464 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.104465 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.104465 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.104563 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:38.104627 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.104882 (+ 255us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.104984 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251026 15:08:38.105185 23675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.095301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33841 (local address 127.14.212.66:46783)
1026 15:08:38.095588 (+ 287us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.095615 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.096425 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.096715 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.096720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.097233 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:38.097669 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.097675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.098742 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.098744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.099519 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.099522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.099684 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.100316 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:38.100338 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:38.102269 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:38.103934 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:38.103940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:38.103943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:38.104156 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:38.104326 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:38.104328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:38.104329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:38.104405 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:38.104625 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:38.104631 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:38.104756 (+ 125us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.104856 (+ 100us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.104994 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":97,"threads_started":1}
I20251026 15:08:38.105640 23782 heartbeater.cc:344] Connected to a master server at 127.14.212.126:33841
I20251026 15:08:38.105726 23782 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:38.105919 23782 heartbeater.cc:507] Master 127.14.212.126:33841 requested a full tablet report, sending...
I20251026 15:08:38.106294 23434 ts_manager.cc:194] Registered new tserver with Master: 716583253ac44e71b7c36c57e3720a0c (127.14.212.66:35369)
I20251026 15:08:38.106789 23434 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:46783
I20251026 15:08:38.109936 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 23651
I20251026 15:08:38.110046 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:08:38.110234 15185 external_mini_cluster.cc:1468] Setting key 613cfee010b06d3df51305b2c7e97fc5
I20251026 15:08:38.111531 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.106694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:39735 (local address 127.14.212.126:33841)
1026 15:08:38.106740 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.106742 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.106821 (+ 79us) server_negotiation.cc:408] Connection header received
1026 15:08:38.106851 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.106851 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.106861 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.106922 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:38.107379 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.107829 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.108315 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.108497 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.109161 (+ 664us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:38.109165 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:38.109165 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:38.109170 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:38.110621 (+ 1451us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.110890 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.110891 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.110891 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.110915 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.111097 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.111098 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.111098 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.111199 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:38.111250 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.111407 (+ 157us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.111471 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251026 15:08:38.111670 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.106621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33841 (local address 127.14.212.66:39735)
1026 15:08:38.106746 (+ 125us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.106756 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.106815 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.106971 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.106972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.107133 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:38.107304 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.107306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.107899 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.107900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.108249 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.108250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.108313 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.108803 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:38.108806 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:38.109057 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:38.110685 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:38.110686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:38.110686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:38.110827 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:38.110984 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:38.110985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:38.110986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:38.111026 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:38.111259 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:38.111259 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:38.111314 (+ 55us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.111391 (+ 77us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.111515 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab.
I20251026 15:08:38.156803 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33841
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:38.240694 23793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:38.240892 23793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:38.240918 23793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:38.242457 23793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:38.242515 23793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:38.242537 23793 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251026 15:08:38.242560 23793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:08:38.244175 23793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.212.126:33841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:38.244415 23793 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:38.244599 23793 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:38.247274 23799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:38.247290 23801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:38.247474 23793 server_base.cc:1047] running on GCE node
W20251026 15:08:38.247380 23798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:38.247681 23793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:38.247881 23793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:38.249012 23793 hybrid_clock.cc:648] HybridClock initialized: now 1761491318248999 us; error 30 us; skew 500 ppm
Oct 26 15:08:38 dist-test-slave-8pgz krb5kdc[23387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491318, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:38.251412 23793 init.cc:377] Logged in from keytab as oryx/127.14.212.67@KRBTEST.COM (short username oryx)
I20251026 15:08:38.251929 23793 webserver.cc:492] Webserver started at http://127.14.212.67:34425/ using document root <none> and password file <none>
I20251026 15:08:38.252111 23793 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:38.252161 23793 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:38.252285 23793 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:38.253428 23793 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "efa6dfe044714b89b53be407cf256c80"
format_stamp: "Formatted at 2025-10-26 15:08:38 on dist-test-slave-8pgz"
server_key: "291f44e20e28c83887f6c0a4969fcce5"
server_key_iv: "6d9a7a0fb3e336234676dbdf795bff9c"
server_key_version: "encryptionkey@0"
I20251026 15:08:38.253756 23793 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "efa6dfe044714b89b53be407cf256c80"
format_stamp: "Formatted at 2025-10-26 15:08:38 on dist-test-slave-8pgz"
server_key: "291f44e20e28c83887f6c0a4969fcce5"
server_key_iv: "6d9a7a0fb3e336234676dbdf795bff9c"
server_key_version: "encryptionkey@0"
I20251026 15:08:38.255088 23793 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:08:38.255755 23808 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:38.255956 23793 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:38.256083 23793 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "efa6dfe044714b89b53be407cf256c80"
format_stamp: "Formatted at 2025-10-26 15:08:38 on dist-test-slave-8pgz"
server_key: "291f44e20e28c83887f6c0a4969fcce5"
server_key_iv: "6d9a7a0fb3e336234676dbdf795bff9c"
server_key_version: "encryptionkey@0"
I20251026 15:08:38.256143 23793 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:38.263489 23793 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:38.265570 23793 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:38.265713 23793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:38.266285 23793 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:38.266335 23793 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:38.266378 23793 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:38.266430 23793 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 26 15:08:38 dist-test-slave-8pgz krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491318, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
I20251026 15:08:38.273696 23793 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:42769
I20251026 15:08:38.273762 23923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:42769 every 8 connection(s)
I20251026 15:08:38.274067 23793 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:08:38.275380 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.266650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:37995 (local address 127.14.212.126:33841)
1026 15:08:38.266725 (+ 75us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.266728 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.266900 (+ 172us) server_negotiation.cc:408] Connection header received
1026 15:08:38.267406 (+ 506us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.267407 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.267423 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.267507 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:38.268336 (+ 829us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.269173 (+ 837us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.269795 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.269974 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.272240 (+ 2266us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:38.272246 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:38.272246 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:38.272256 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:38.274029 (+ 1773us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.274590 (+ 561us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.274591 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.274591 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.274615 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.274778 (+ 163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.274778 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.274778 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.274870 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:38.274918 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.275207 (+ 289us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.275299 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251026 15:08:38.275483 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.266626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33841 (local address 127.14.212.67:37995)
1026 15:08:38.266789 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.266819 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.267302 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.267566 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.267569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.267937 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:38.268234 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.268238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.269294 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.269295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.269705 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.269707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.269852 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.270539 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:38.270564 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:38.272134 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:38.274112 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:38.274117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:38.274119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:38.274500 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:38.274677 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:38.274677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:38.274678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:38.274732 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:38.274950 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:38.274956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:38.275088 (+ 132us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.275193 (+ 105us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.275332 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20251026 15:08:38.276017 23924 heartbeater.cc:344] Connected to a master server at 127.14.212.126:33841
I20251026 15:08:38.276103 23924 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:38.276333 23924 heartbeater.cc:507] Master 127.14.212.126:33841 requested a full tablet report, sending...
I20251026 15:08:38.276732 23435 ts_manager.cc:194] Registered new tserver with Master: efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769)
I20251026 15:08:38.277174 23435 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:37995
I20251026 15:08:38.281716 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 23793
I20251026 15:08:38.281832 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:08:38.281886 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.277078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33841 (local address 127.14.212.67:40499)
1026 15:08:38.277260 (+ 182us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.277270 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.277334 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.277475 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.277476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.277617 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:38.277797 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.277799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.278465 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.278466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.278820 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.278822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.278881 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.279392 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:38.279396 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:38.279658 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:38.281119 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:38.281120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:38.281121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:38.281260 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:38.281396 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:38.281397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:38.281397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:38.281417 (+ 20us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:38.281600 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:38.281601 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:38.281639 (+ 38us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.281687 (+ 48us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.281763 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":67,"threads_started":1}
I20251026 15:08:38.281922 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.277189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:40499 (local address 127.14.212.126:33841)
1026 15:08:38.277233 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.277235 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.277317 (+ 82us) server_negotiation.cc:408] Connection header received
1026 15:08:38.277404 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.277405 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.277414 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.277464 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:38.277872 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.278356 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.278883 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.278989 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.279741 (+ 752us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:38.279744 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:38.279744 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:38.279750 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:38.281012 (+ 1262us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.281330 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.281331 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.281331 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.281349 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.281456 (+ 107us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.281457 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.281457 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.281549 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:38.281616 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.281729 (+ 113us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.281814 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20251026 15:08:38.282038 15185 external_mini_cluster.cc:1468] Setting key 03356ec82402e212addcea8ebcb5e6cf
I20251026 15:08:38.282946 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 26 15:08:38 dist-test-slave-8pgz krb5kdc[23387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 26 15:08:38 dist-test-slave-8pgz krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:38 dist-test-slave-8pgz krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
W20251026 15:08:38.302517 23504 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:38.297533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.14.212.126:33841)
1026 15:08:38.297600 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.297602 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.297612 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:38.297638 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.297639 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.297648 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.297708 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:38.298382 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.298948 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.299532 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.299650 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.302438 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37512: BlockingRecv error: recv got EOF from 127.0.0.1:37512 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Oct 26 15:08:38 dist-test-slave-8pgz krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
I20251026 15:08:38.309381 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.303403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.14.212.126:33841)
1026 15:08:38.303454 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.303455 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.303508 (+ 53us) server_negotiation.cc:408] Connection header received
1026 15:08:38.303562 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.303563 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.303570 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.303623 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:38.304158 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.304610 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.305198 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.305318 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.307361 (+ 2043us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:38.307365 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:38.307365 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:38.307372 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:38.308611 (+ 1239us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.308885 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.308885 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.308886 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.308906 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.309004 (+ 98us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.309004 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.309004 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.309105 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:38.309181 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.309252 (+ 71us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.309312 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251026 15:08:38.311055 23435 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:08:38.312367 23435 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:08:38.317317 23947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.314500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:42769 (local address 127.0.0.1:34598)
1026 15:08:38.314656 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.314666 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.314736 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.315031 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.315034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.315037 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:38.315260 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.315262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.316358 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.316359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.317031 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.317032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.317114 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.317154 (+ 40us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.317203 (+ 49us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.317240 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20251026 15:08:38.317785 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.314734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.14.212.67:42769)
1026 15:08:38.314877 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.314883 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.314892 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:08:38.314923 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.314924 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.314937 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.314998 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:38.315365 (+ 367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.316279 (+ 914us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.317157 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.317581 (+ 424us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.317627 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.317661 (+ 34us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.317709 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20251026 15:08:38.317962 23946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.314485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:35369 (local address 127.0.0.1:41574)
1026 15:08:38.314632 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.314659 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.314734 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.315293 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.315293 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.315296 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:38.315529 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.315531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.316872 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.316873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.317715 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.317717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.317793 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.317797 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.317831 (+ 34us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.317876 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":46,"threads_started":1}
I20251026 15:08:38.318647 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.314689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.14.212.66:35369)
1026 15:08:38.314943 (+ 254us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.314951 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.314962 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:08:38.315002 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.315004 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.315019 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.315274 (+ 255us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:38.315611 (+ 337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.316783 (+ 1172us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.317817 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.318504 (+ 687us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.318539 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.318565 (+ 26us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.318592 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":86,"threads_started":1}
I20251026 15:08:38.318673 23858 tablet_service.cc:1505] Processing CreateTablet for tablet d6821e1d34cb4debb2685fa72b12222c (DEFAULT_TABLE table=test-table [id=4414d7281d9a4d24918b3aeabcd00350]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:38.318987 23858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6821e1d34cb4debb2685fa72b12222c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.319103 23716 tablet_service.cc:1505] Processing CreateTablet for tablet d6821e1d34cb4debb2685fa72b12222c (DEFAULT_TABLE table=test-table [id=4414d7281d9a4d24918b3aeabcd00350]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:38.319237 23948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.314545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:41571 (local address 127.0.0.1:59958)
1026 15:08:38.315121 (+ 576us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.315132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.315185 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.315429 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.315429 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.315431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:38.315592 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.315593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.318151 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.318152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.319021 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.319023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.319107 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.319111 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.319143 (+ 32us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.319182 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":76,"threads_started":1}
I20251026 15:08:38.319394 23716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6821e1d34cb4debb2685fa72b12222c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.320103 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.314921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.14.212.65:41571)
1026 15:08:38.315073 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.315081 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.315196 (+ 115us) server_negotiation.cc:408] Connection header received
1026 15:08:38.315243 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.315246 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.315262 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.315323 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:38.315684 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.316800 (+ 1116us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.319371 (+ 2571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.319892 (+ 521us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.319944 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.319979 (+ 35us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.320025 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20251026 15:08:38.321631 23953 tablet_bootstrap.cc:492] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c: Bootstrap starting.
I20251026 15:08:38.321837 23952 tablet_bootstrap.cc:492] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80: Bootstrap starting.
I20251026 15:08:38.322216 23953 tablet_bootstrap.cc:654] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.322122 23574 tablet_service.cc:1505] Processing CreateTablet for tablet d6821e1d34cb4debb2685fa72b12222c (DEFAULT_TABLE table=test-table [id=4414d7281d9a4d24918b3aeabcd00350]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:38.322443 23574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6821e1d34cb4debb2685fa72b12222c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.322526 23952 tablet_bootstrap.cc:654] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.322860 23953 log.cc:826] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:38.323246 23952 log.cc:826] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:38.324007 23952 tablet_bootstrap.cc:492] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80: No bootstrap required, opened a new log
I20251026 15:08:38.324007 23953 tablet_bootstrap.cc:492] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c: No bootstrap required, opened a new log
I20251026 15:08:38.324079 23952 ts_tablet_manager.cc:1403] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251026 15:08:38.324131 23953 ts_tablet_manager.cc:1403] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251026 15:08:38.325037 23956 tablet_bootstrap.cc:492] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0: Bootstrap starting.
I20251026 15:08:38.325624 23956 tablet_bootstrap.cc:654] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.325646 23953 raft_consensus.cc:359] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } }
I20251026 15:08:38.325641 23952 raft_consensus.cc:359] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } }
I20251026 15:08:38.325790 23952 raft_consensus.cc:385] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.325790 23953 raft_consensus.cc:385] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.325829 23953 raft_consensus.cc:740] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716583253ac44e71b7c36c57e3720a0c, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.325829 23952 raft_consensus.cc:740] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa6dfe044714b89b53be407cf256c80, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.325918 23952 consensus_queue.cc:260] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } }
I20251026 15:08:38.325938 23953 consensus_queue.cc:260] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } }
I20251026 15:08:38.326157 23953 ts_tablet_manager.cc:1434] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:08:38.326157 23952 ts_tablet_manager.cc:1434] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:08:38.326274 23782 heartbeater.cc:499] Master 127.14.212.126:33841 was elected leader, sending a full tablet report...
I20251026 15:08:38.326279 23924 heartbeater.cc:499] Master 127.14.212.126:33841 was elected leader, sending a full tablet report...
I20251026 15:08:38.326259 23956 log.cc:826] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:38.327194 23956 tablet_bootstrap.cc:492] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0: No bootstrap required, opened a new log
I20251026 15:08:38.327255 23956 ts_tablet_manager.cc:1403] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:08:38.328573 23956 raft_consensus.cc:359] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } }
I20251026 15:08:38.328692 23956 raft_consensus.cc:385] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.328720 23956 raft_consensus.cc:740] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9473e4ac53945b2ae886fd889a8cef0, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.328822 23956 consensus_queue.cc:260] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } }
I20251026 15:08:38.329032 23956 ts_tablet_manager.cc:1434] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:08:38.329088 23640 heartbeater.cc:499] Master 127.14.212.126:33841 was elected leader, sending a full tablet report...
W20251026 15:08:38.352859 23783 tablet.cc:2378] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:08:38.406304 23958 raft_consensus.cc:493] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:08:38.406484 23958 raft_consensus.cc:515] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } }
I20251026 15:08:38.406888 23958 leader_election.cc:290] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769), e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571)
I20251026 15:08:38.409497 23675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.407080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:41571 (local address 127.14.212.66:45075)
1026 15:08:38.407133 (+ 53us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.407147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.407199 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.407360 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.407361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.407363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:38.407561 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.407563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.408469 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.408470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.409264 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.409266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.409351 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.409355 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.409386 (+ 31us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.409425 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20251026 15:08:38.409708 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.407087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:42769 (local address 127.14.212.66:47825)
1026 15:08:38.407305 (+ 218us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.407315 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.407390 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.407536 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.407537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.407538 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:38.407796 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.407798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.408675 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.408676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.409494 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.409496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.409561 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.409565 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.409596 (+ 31us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.409641 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":12,"thread_start_us":110,"threads_started":1}
I20251026 15:08:38.409993 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.407178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:45075 (local address 127.14.212.65:41571)
1026 15:08:38.407229 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.407231 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.407241 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:38.407271 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.407272 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.407281 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.407352 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:38.407641 (+ 289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.408379 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.409359 (+ 980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.409859 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.409869 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.409891 (+ 22us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.409927 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251026 15:08:38.410192 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.407217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:47825 (local address 127.14.212.67:42769)
1026 15:08:38.407283 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.407286 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.407388 (+ 102us) server_negotiation.cc:408] Connection header received
1026 15:08:38.407457 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.407458 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.407468 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.407523 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:38.407889 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.408601 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.409593 (+ 992us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.410070 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.410079 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.410102 (+ 23us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.410131 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251026 15:08:38.410429 23594 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6821e1d34cb4debb2685fa72b12222c" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9473e4ac53945b2ae886fd889a8cef0" is_pre_election: true
I20251026 15:08:38.410570 23594 raft_consensus.cc:2468] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 0.
I20251026 15:08:38.410506 23877 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6821e1d34cb4debb2685fa72b12222c" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa6dfe044714b89b53be407cf256c80" is_pre_election: true
I20251026 15:08:38.410671 23877 raft_consensus.cc:2468] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 0.
I20251026 15:08:38.410771 23668 leader_election.cc:304] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716583253ac44e71b7c36c57e3720a0c, e9473e4ac53945b2ae886fd889a8cef0; no voters:
I20251026 15:08:38.410931 23958 raft_consensus.cc:2804] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:08:38.410988 23958 raft_consensus.cc:493] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:08:38.411007 23958 raft_consensus.cc:3060] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.411715 23958 raft_consensus.cc:515] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } }
I20251026 15:08:38.411834 23958 leader_election.cc:290] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 election: Requested vote from peers efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769), e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571)
I20251026 15:08:38.412014 23594 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6821e1d34cb4debb2685fa72b12222c" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9473e4ac53945b2ae886fd889a8cef0"
I20251026 15:08:38.412096 23594 raft_consensus.cc:3060] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.412779 23594 raft_consensus.cc:2468] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 1.
I20251026 15:08:38.412891 23877 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6821e1d34cb4debb2685fa72b12222c" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa6dfe044714b89b53be407cf256c80"
I20251026 15:08:38.412966 23668 leader_election.cc:304] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716583253ac44e71b7c36c57e3720a0c, e9473e4ac53945b2ae886fd889a8cef0; no voters:
I20251026 15:08:38.412967 23877 raft_consensus.cc:3060] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.413075 23958 raft_consensus.cc:2804] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:38.413305 23958 raft_consensus.cc:697] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [term 1 LEADER]: Becoming Leader. State: Replica: 716583253ac44e71b7c36c57e3720a0c, State: Running, Role: LEADER
I20251026 15:08:38.413414 23958 consensus_queue.cc:237] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } }
I20251026 15:08:38.413812 23877 raft_consensus.cc:2468] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 1.
I20251026 15:08:38.414211 23434 catalog_manager.cc:5649] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 716583253ac44e71b7c36c57e3720a0c (127.14.212.66). New cstate: current_term: 1 leader_uuid: "716583253ac44e71b7c36c57e3720a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } health_report { overall_health: UNKNOWN } } }
W20251026 15:08:38.430256 23641 tablet.cc:2378] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:08:38.435406 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.433282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.14.212.66:35369)
1026 15:08:38.433333 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.433335 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.433348 (+ 13us) server_negotiation.cc:408] Connection header received
1026 15:08:38.433443 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.433444 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.433461 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.433514 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:08:38.433728 (+ 214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.434238 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.434880 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.435007 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.435038 (+ 31us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:08:38.435156 (+ 118us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:08:38.435220 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.435313 (+ 93us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.435339 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251026 15:08:38.437163 23594 raft_consensus.cc:1275] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Refusing update from remote peer 716583253ac44e71b7c36c57e3720a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:38.437175 23877 raft_consensus.cc:1275] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Refusing update from remote peer 716583253ac44e71b7c36c57e3720a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:38.437459 23958 consensus_queue.cc:1048] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:38.437577 23962 consensus_queue.cc:1048] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:38.439859 23965 mvcc.cc:204] Tried to move back new op lower bound from 7215068440315973632 to 7215068440221790208. Current Snapshot: MvccSnapshot[applied={T|T < 7215068440315973632}]
I20251026 15:08:38.440198 23966 mvcc.cc:204] Tried to move back new op lower bound from 7215068440315973632 to 7215068440221790208. Current Snapshot: MvccSnapshot[applied={T|T < 7215068440315973632}]
I20251026 15:08:38.441440 23968 mvcc.cc:204] Tried to move back new op lower bound from 7215068440315973632 to 7215068440221790208. Current Snapshot: MvccSnapshot[applied={T|T < 7215068440315973632 or (T in {7215068440315973632})}]
I20251026 15:08:38.442799 23434 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:38.442888 23434 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:38.443614 23434 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a: Sending DeleteTablet for 3 replicas of tablet d6821e1d34cb4debb2685fa72b12222c
I20251026 15:08:38.443893 23716 tablet_service.cc:1552] Processing DeleteTablet for tablet d6821e1d34cb4debb2685fa72b12222c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:38 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:41574
I20251026 15:08:38.444017 23858 tablet_service.cc:1552] Processing DeleteTablet for tablet d6821e1d34cb4debb2685fa72b12222c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:38 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:34598
I20251026 15:08:38.443995 23574 tablet_service.cc:1552] Processing DeleteTablet for tablet d6821e1d34cb4debb2685fa72b12222c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:38 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:59958
I20251026 15:08:38.444205 23975 tablet_replica.cc:333] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0: stopping tablet replica
I20251026 15:08:38.444299 23975 raft_consensus.cc:2243] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:38.444337 23434 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 15:08:38.444416 23975 raft_consensus.cc:2272] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 15:08:38.444478 23434 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:08:38.444813 23974 tablet_replica.cc:333] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c: stopping tablet replica
I20251026 15:08:38.444878 23975 ts_tablet_manager.cc:1916] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:38.446171 23574 tablet_service.cc:1505] Processing CreateTablet for tablet 31106fd8f70345bd842ebe7913e38dbc (DEFAULT_TABLE table=test-table [id=67d90592b26747a49ae106e15b603379]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:38.446213 23858 tablet_service.cc:1505] Processing CreateTablet for tablet 31106fd8f70345bd842ebe7913e38dbc (DEFAULT_TABLE table=test-table [id=67d90592b26747a49ae106e15b603379]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:38.446283 23574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31106fd8f70345bd842ebe7913e38dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.446321 23858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31106fd8f70345bd842ebe7913e38dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.446666 23975 ts_tablet_manager.cc:1929] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:08:38.446724 23975 log.cc:1199] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/wal/wals/d6821e1d34cb4debb2685fa72b12222c
I20251026 15:08:38.446956 23975 ts_tablet_manager.cc:1950] T d6821e1d34cb4debb2685fa72b12222c P e9473e4ac53945b2ae886fd889a8cef0: Deleting consensus metadata
I20251026 15:08:38.447340 23976 tablet_replica.cc:333] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80: stopping tablet replica
I20251026 15:08:38.447428 23976 raft_consensus.cc:2243] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:38.447520 23976 raft_consensus.cc:2272] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:38.447579 23956 tablet_bootstrap.cc:492] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0: Bootstrap starting.
I20251026 15:08:38.447790 23952 tablet_bootstrap.cc:492] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80: Bootstrap starting.
I20251026 15:08:38.447968 23956 tablet_bootstrap.cc:654] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.448141 23420 catalog_manager.cc:4985] TS e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571): tablet d6821e1d34cb4debb2685fa72b12222c (table test-table [id=4414d7281d9a4d24918b3aeabcd00350]) successfully deleted
I20251026 15:08:38.448238 23952 tablet_bootstrap.cc:654] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.448477 23716 tablet_service.cc:1505] Processing CreateTablet for tablet 31106fd8f70345bd842ebe7913e38dbc (DEFAULT_TABLE table=test-table [id=67d90592b26747a49ae106e15b603379]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:38.448783 23974 raft_consensus.cc:2243] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:08:38.448824 23716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31106fd8f70345bd842ebe7913e38dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.448940 23974 raft_consensus.cc:2272] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:38.449347 23976 ts_tablet_manager.cc:1916] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:38.450276 23953 tablet_bootstrap.cc:492] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c: Bootstrap starting.
I20251026 15:08:38.450783 23953 tablet_bootstrap.cc:654] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.451025 23952 tablet_bootstrap.cc:492] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80: No bootstrap required, opened a new log
I20251026 15:08:38.451069 23952 ts_tablet_manager.cc:1403] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20251026 15:08:38.451263 23976 ts_tablet_manager.cc:1929] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:08:38.451233 23952 raft_consensus.cc:359] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.451323 23952 raft_consensus.cc:385] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.451349 23952 raft_consensus.cc:740] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa6dfe044714b89b53be407cf256c80, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.451396 23952 consensus_queue.cc:260] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.451491 23952 ts_tablet_manager.cc:1434] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 15:08:38.451530 23974 ts_tablet_manager.cc:1916] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:38.451849 23953 tablet_bootstrap.cc:492] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c: No bootstrap required, opened a new log
I20251026 15:08:38.451900 23953 ts_tablet_manager.cc:1403] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 15:08:38.452052 23976 log.cc:1199] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/d6821e1d34cb4debb2685fa72b12222c
I20251026 15:08:38.452045 23953 raft_consensus.cc:359] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.452106 23953 raft_consensus.cc:385] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.452128 23953 raft_consensus.cc:740] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716583253ac44e71b7c36c57e3720a0c, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.452190 23953 consensus_queue.cc:260] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.452266 23953 ts_tablet_manager.cc:1434] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:38.452337 23976 ts_tablet_manager.cc:1950] T d6821e1d34cb4debb2685fa72b12222c P efa6dfe044714b89b53be407cf256c80: Deleting consensus metadata
I20251026 15:08:38.452746 23422 catalog_manager.cc:4985] TS efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769): tablet d6821e1d34cb4debb2685fa72b12222c (table test-table [id=4414d7281d9a4d24918b3aeabcd00350]) successfully deleted
I20251026 15:08:38.453385 23974 ts_tablet_manager.cc:1929] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:08:38.453449 23974 log.cc:1199] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/d6821e1d34cb4debb2685fa72b12222c
I20251026 15:08:38.453548 23958 raft_consensus.cc:493] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:08:38.453600 23958 raft_consensus.cc:515] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.453722 23974 ts_tablet_manager.cc:1950] T d6821e1d34cb4debb2685fa72b12222c P 716583253ac44e71b7c36c57e3720a0c: Deleting consensus metadata
I20251026 15:08:38.453743 23958 leader_election.cc:290] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571), efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769)
I20251026 15:08:38.453925 23877 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31106fd8f70345bd842ebe7913e38dbc" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa6dfe044714b89b53be407cf256c80" is_pre_election: true
I20251026 15:08:38.453980 23877 raft_consensus.cc:2468] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 0.
I20251026 15:08:38.453927 23594 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31106fd8f70345bd842ebe7913e38dbc" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9473e4ac53945b2ae886fd889a8cef0" is_pre_election: true
I20251026 15:08:38.454151 23670 leader_election.cc:304] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716583253ac44e71b7c36c57e3720a0c, efa6dfe044714b89b53be407cf256c80; no voters:
I20251026 15:08:38.454249 23958 raft_consensus.cc:2804] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251026 15:08:38.454236 23668 leader_election.cc:343] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571): Illegal state: must be running to vote when last-logged opid is not known
I20251026 15:08:38.454304 23958 raft_consensus.cc:493] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:08:38.454319 23958 raft_consensus.cc:3060] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.454424 23421 catalog_manager.cc:4985] TS 716583253ac44e71b7c36c57e3720a0c (127.14.212.66:35369): tablet d6821e1d34cb4debb2685fa72b12222c (table test-table [id=4414d7281d9a4d24918b3aeabcd00350]) successfully deleted
I20251026 15:08:38.454913 23958 raft_consensus.cc:515] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.455029 23958 leader_election.cc:290] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 election: Requested vote from peers e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571), efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769)
I20251026 15:08:38.455224 23877 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31106fd8f70345bd842ebe7913e38dbc" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa6dfe044714b89b53be407cf256c80"
I20251026 15:08:38.455309 23877 raft_consensus.cc:3060] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.456087 23877 raft_consensus.cc:2468] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 1.
I20251026 15:08:38.456274 23594 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31106fd8f70345bd842ebe7913e38dbc" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9473e4ac53945b2ae886fd889a8cef0"
I20251026 15:08:38.456333 23670 leader_election.cc:304] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716583253ac44e71b7c36c57e3720a0c, efa6dfe044714b89b53be407cf256c80; no voters:
I20251026 15:08:38.456451 23958 raft_consensus.cc:2804] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Leader election won for term 1
W20251026 15:08:38.456480 23668 leader_election.cc:343] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571): Illegal state: must be running to vote when last-logged opid is not known
I20251026 15:08:38.456517 23958 raft_consensus.cc:697] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [term 1 LEADER]: Becoming Leader. State: Replica: 716583253ac44e71b7c36c57e3720a0c, State: Running, Role: LEADER
I20251026 15:08:38.456560 23958 consensus_queue.cc:237] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.457098 23434 catalog_manager.cc:5649] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 716583253ac44e71b7c36c57e3720a0c (127.14.212.66). New cstate: current_term: 1 leader_uuid: "716583253ac44e71b7c36c57e3720a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } health_report { overall_health: UNKNOWN } } }
I20251026 15:08:38.458853 23956 tablet_bootstrap.cc:492] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0: No bootstrap required, opened a new log
I20251026 15:08:38.458910 23956 ts_tablet_manager.cc:1403] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20251026 15:08:38.459087 23956 raft_consensus.cc:359] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.459170 23956 raft_consensus.cc:385] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.459185 23956 raft_consensus.cc:740] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9473e4ac53945b2ae886fd889a8cef0, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.459224 23956 consensus_queue.cc:260] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.459323 23956 ts_tablet_manager.cc:1434] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
Oct 26 15:08:38 dist-test-slave-8pgz krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491317, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.126@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
I20251026 15:08:38.486246 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.479391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.14.212.126:33841)
1026 15:08:38.479442 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.479445 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.479581 (+ 136us) server_negotiation.cc:408] Connection header received
1026 15:08:38.479657 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.479658 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.479668 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.479722 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:38.480383 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.480851 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.481385 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.481546 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.483678 (+ 2132us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:38.483685 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:38.483685 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:38.483692 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:38.485208 (+ 1516us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.485572 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.485573 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.485574 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.485597 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.485729 (+ 132us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.485730 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.485730 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.485822 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:38.485906 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.486018 (+ 112us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.486108 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251026 15:08:38.486323 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.479333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33841 (local address 127.0.0.1:37522)
1026 15:08:38.479504 (+ 171us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.479514 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.479592 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.479731 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.479732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.480117 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:38.480281 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.480284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.480924 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.480925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.481307 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.481309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.481379 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.482099 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:38.482103 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:38.483594 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:38.485301 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:38.485309 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:38.485312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:38.485504 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:38.485651 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:38.485651 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:38.485652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:38.485679 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:38.485874 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:38.485880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:38.485926 (+ 46us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.485993 (+ 67us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.486178 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":76,"thread_start_us":112,"threads_started":1}
I20251026 15:08:38.488704 23433 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:37522:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251026 15:08:38.488948 23433 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:08:38.490520 23574 tablet_service.cc:1505] Processing CreateTablet for tablet 1b17f0bd866845ebbe1ce945c38b7d9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c0a50d0a5a443eba9e43e3850679757]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 15:08:38.490522 23716 tablet_service.cc:1505] Processing CreateTablet for tablet 1b17f0bd866845ebbe1ce945c38b7d9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c0a50d0a5a443eba9e43e3850679757]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 15:08:38.490633 23858 tablet_service.cc:1505] Processing CreateTablet for tablet 1b17f0bd866845ebbe1ce945c38b7d9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c0a50d0a5a443eba9e43e3850679757]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251026 15:08:38.490694 23574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b17f0bd866845ebbe1ce945c38b7d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.490695 23716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b17f0bd866845ebbe1ce945c38b7d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.490747 23858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b17f0bd866845ebbe1ce945c38b7d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.492187 23956 tablet_bootstrap.cc:492] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0: Bootstrap starting.
I20251026 15:08:38.492187 23952 tablet_bootstrap.cc:492] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80: Bootstrap starting.
I20251026 15:08:38.492624 23952 tablet_bootstrap.cc:654] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.492805 23956 tablet_bootstrap.cc:654] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.493139 23953 tablet_bootstrap.cc:492] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c: Bootstrap starting.
I20251026 15:08:38.493391 23952 tablet_bootstrap.cc:492] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80: No bootstrap required, opened a new log
I20251026 15:08:38.493441 23952 ts_tablet_manager.cc:1403] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:38.493619 23952 raft_consensus.cc:359] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.493655 23953 tablet_bootstrap.cc:654] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.493678 23952 raft_consensus.cc:385] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.493703 23952 raft_consensus.cc:740] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa6dfe044714b89b53be407cf256c80, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.493752 23952 consensus_queue.cc:260] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.493854 23952 ts_tablet_manager.cc:1434] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 15:08:38.494007 23957 tablet_replica.cc:442] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } }
I20251026 15:08:38.494104 23957 tablet_replica.cc:445] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:08:38.494565 23953 tablet_bootstrap.cc:492] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c: No bootstrap required, opened a new log
I20251026 15:08:38.494596 23956 tablet_bootstrap.cc:492] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0: No bootstrap required, opened a new log
I20251026 15:08:38.494621 23953 ts_tablet_manager.cc:1403] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:38.494642 23956 ts_tablet_manager.cc:1403] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 15:08:38.494794 23953 raft_consensus.cc:359] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.494801 23956 raft_consensus.cc:359] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.494863 23956 raft_consensus.cc:385] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.494863 23953 raft_consensus.cc:385] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.494884 23956 raft_consensus.cc:740] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9473e4ac53945b2ae886fd889a8cef0, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.494887 23953 raft_consensus.cc:740] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716583253ac44e71b7c36c57e3720a0c, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.494925 23956 consensus_queue.cc:260] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.495013 23956 ts_tablet_manager.cc:1434] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:38.494941 23953 consensus_queue.cc:260] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.495198 23953 ts_tablet_manager.cc:1434] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:38.495185 23960 tablet_replica.cc:442] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } }
I20251026 15:08:38.495275 23960 tablet_replica.cc:445] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:08:38.495404 23962 tablet_replica.cc:442] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } }
I20251026 15:08:38.495497 23962 tablet_replica.cc:445] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:08:38.524149 23962 raft_consensus.cc:493] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:08:38.524252 23962 raft_consensus.cc:515] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.524396 23962 leader_election.cc:290] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571), efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769)
W20251026 15:08:38.524639 23925 tablet.cc:2378] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:08:38.524662 23877 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b17f0bd866845ebbe1ce945c38b7d9f" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa6dfe044714b89b53be407cf256c80" is_pre_election: true
I20251026 15:08:38.524729 23594 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b17f0bd866845ebbe1ce945c38b7d9f" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9473e4ac53945b2ae886fd889a8cef0" is_pre_election: true
I20251026 15:08:38.524785 23877 raft_consensus.cc:2468] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 0.
I20251026 15:08:38.524808 23594 raft_consensus.cc:2468] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 0.
I20251026 15:08:38.524961 23670 leader_election.cc:304] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716583253ac44e71b7c36c57e3720a0c, efa6dfe044714b89b53be407cf256c80; no voters:
I20251026 15:08:38.525130 23962 raft_consensus.cc:2804] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:08:38.525194 23962 raft_consensus.cc:493] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:08:38.525230 23962 raft_consensus.cc:3060] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.526088 23962 raft_consensus.cc:515] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.526237 23962 leader_election.cc:290] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 election: Requested vote from peers e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571), efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769)
I20251026 15:08:38.526419 23594 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b17f0bd866845ebbe1ce945c38b7d9f" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9473e4ac53945b2ae886fd889a8cef0"
I20251026 15:08:38.526466 23877 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b17f0bd866845ebbe1ce945c38b7d9f" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa6dfe044714b89b53be407cf256c80"
I20251026 15:08:38.526499 23594 raft_consensus.cc:3060] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.526552 23877 raft_consensus.cc:3060] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.527145 23594 raft_consensus.cc:2468] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 1.
I20251026 15:08:38.527338 23877 raft_consensus.cc:2468] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 1.
I20251026 15:08:38.527350 23668 leader_election.cc:304] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716583253ac44e71b7c36c57e3720a0c, e9473e4ac53945b2ae886fd889a8cef0; no voters:
I20251026 15:08:38.527508 23962 raft_consensus.cc:2804] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:38.527575 23962 raft_consensus.cc:697] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [term 1 LEADER]: Becoming Leader. State: Replica: 716583253ac44e71b7c36c57e3720a0c, State: Running, Role: LEADER
I20251026 15:08:38.527626 23962 consensus_queue.cc:237] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.527972 23958 tablet_replica.cc:442] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c: TxnStatusTablet state changed. Reason: New leader 716583253ac44e71b7c36c57e3720a0c. Latest consensus state: current_term: 1 leader_uuid: "716583253ac44e71b7c36c57e3720a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } }
I20251026 15:08:38.528062 23958 tablet_replica.cc:445] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c: This TxnStatusTablet replica's current role is: LEADER
I20251026 15:08:38.528306 23991 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251026 15:08:38.528272 23433 catalog_manager.cc:5649] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 716583253ac44e71b7c36c57e3720a0c (127.14.212.66). New cstate: current_term: 1 leader_uuid: "716583253ac44e71b7c36c57e3720a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } health_report { overall_health: UNKNOWN } } }
I20251026 15:08:38.528410 23991 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251026 15:08:38.534286 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.532003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.14.212.66:35369)
1026 15:08:38.532064 (+ 61us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.532066 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.532187 (+ 121us) server_negotiation.cc:408] Connection header received
1026 15:08:38.532249 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.532250 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.532260 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.532312 (+ 52us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:08:38.532655 (+ 343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.533125 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.533889 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.534015 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.534032 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:08:38.534056 (+ 24us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:08:38.534111 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.534210 (+ 99us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.534230 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251026 15:08:38.534327 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.531934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:35369 (local address 127.0.0.1:41598)
1026 15:08:38.532096 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.532109 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.532177 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.532334 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.532335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.532337 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 15:08:38.532571 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.532574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.533189 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.533190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.533796 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.533799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.533882 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.533902 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 15:08:38.534132 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 15:08:38.534143 (+ 11us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.534196 (+ 53us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.534250 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20251026 15:08:38.535419 23877 raft_consensus.cc:1275] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Refusing update from remote peer 716583253ac44e71b7c36c57e3720a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:38.535435 23594 raft_consensus.cc:1275] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Refusing update from remote peer 716583253ac44e71b7c36c57e3720a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:38.535617 23958 consensus_queue.cc:1048] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:38.535696 23962 consensus_queue.cc:1048] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:38.536203 23957 tablet_replica.cc:442] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80: TxnStatusTablet state changed. Reason: New leader 716583253ac44e71b7c36c57e3720a0c. Latest consensus state: current_term: 1 leader_uuid: "716583253ac44e71b7c36c57e3720a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } }
I20251026 15:08:38.536294 23957 tablet_replica.cc:445] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:08:38.536298 23960 tablet_replica.cc:442] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0: TxnStatusTablet state changed. Reason: New leader 716583253ac44e71b7c36c57e3720a0c. Latest consensus state: current_term: 1 leader_uuid: "716583253ac44e71b7c36c57e3720a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } }
I20251026 15:08:38.536373 23960 tablet_replica.cc:445] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:08:38.537111 23962 tablet_replica.cc:442] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "716583253ac44e71b7c36c57e3720a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } }
I20251026 15:08:38.537187 23962 tablet_replica.cc:445] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c: This TxnStatusTablet replica's current role is: LEADER
I20251026 15:08:38.537325 23958 tablet_replica.cc:442] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "716583253ac44e71b7c36c57e3720a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } }
I20251026 15:08:38.537398 23958 tablet_replica.cc:445] T 1b17f0bd866845ebbe1ce945c38b7d9f P 716583253ac44e71b7c36c57e3720a0c: This TxnStatusTablet replica's current role is: LEADER
I20251026 15:08:38.538021 23960 tablet_replica.cc:442] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "716583253ac44e71b7c36c57e3720a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } }
I20251026 15:08:38.538038 23957 tablet_replica.cc:442] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "716583253ac44e71b7c36c57e3720a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } } }
I20251026 15:08:38.538110 23960 tablet_replica.cc:445] T 1b17f0bd866845ebbe1ce945c38b7d9f P e9473e4ac53945b2ae886fd889a8cef0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:08:38.538118 23957 tablet_replica.cc:445] T 1b17f0bd866845ebbe1ce945c38b7d9f P efa6dfe044714b89b53be407cf256c80: This TxnStatusTablet replica's current role is: FOLLOWER
I20251026 15:08:38.543756 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.541431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:35369 (local address 127.14.212.66:55993)
1026 15:08:38.541527 (+ 96us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.541540 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.541583 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.541719 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.541721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.541722 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 15:08:38.541888 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.541889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.542762 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.542763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.543187 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.543188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.543250 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.543264 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 15:08:38.543558 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 15:08:38.543559 (+ 1us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.543602 (+ 43us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.543646 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20251026 15:08:38.543759 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.541501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:55993 (local address 127.14.212.66:35369)
1026 15:08:38.541557 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.541559 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.541573 (+ 14us) server_negotiation.cc:408] Connection header received
1026 15:08:38.541628 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.541629 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.541639 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.541698 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:08:38.541960 (+ 262us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.542681 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.543269 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.543435 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.543459 (+ 24us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:08:38.543487 (+ 28us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:08:38.543550 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.543616 (+ 66us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.543638 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251026 15:08:38.546229 23594 raft_consensus.cc:3060] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.546276 23877 raft_consensus.cc:1275] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Refusing update from remote peer 716583253ac44e71b7c36c57e3720a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:38.546533 23995 consensus_queue.cc:1048] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:38.547298 23594 raft_consensus.cc:1275] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Refusing update from remote peer 716583253ac44e71b7c36c57e3720a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:38.547533 23995 consensus_queue.cc:1048] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:38.558666 23433 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:38.558775 23433 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:38.559464 23433 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a: Sending DeleteTablet for 3 replicas of tablet 31106fd8f70345bd842ebe7913e38dbc
I20251026 15:08:38.559691 23574 tablet_service.cc:1552] Processing DeleteTablet for tablet 31106fd8f70345bd842ebe7913e38dbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:38 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:59958
I20251026 15:08:38.559799 23975 tablet_replica.cc:333] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0: stopping tablet replica
I20251026 15:08:38.559754 23716 tablet_service.cc:1552] Processing DeleteTablet for tablet 31106fd8f70345bd842ebe7913e38dbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:38 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:41574
I20251026 15:08:38.559799 23858 tablet_service.cc:1552] Processing DeleteTablet for tablet 31106fd8f70345bd842ebe7913e38dbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:38 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:34598
I20251026 15:08:38.559849 23975 raft_consensus.cc:2243] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:38.559878 23976 tablet_replica.cc:333] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80: stopping tablet replica
I20251026 15:08:38.559890 23975 raft_consensus.cc:2272] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:38.559888 23974 tablet_replica.cc:333] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c: stopping tablet replica
I20251026 15:08:38.559931 23974 raft_consensus.cc:2243] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:08:38.559980 23976 raft_consensus.cc:2243] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:38.559986 23974 raft_consensus.cc:2272] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:38.560017 23976 raft_consensus.cc:2272] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:38.560428 23976 ts_tablet_manager.cc:1916] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:38.560597 23974 ts_tablet_manager.cc:1916] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:38.562388 23974 ts_tablet_manager.cc:1929] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:08:38.562465 23974 log.cc:1199] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/31106fd8f70345bd842ebe7913e38dbc
I20251026 15:08:38.562543 23975 ts_tablet_manager.cc:1916] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:38.562678 23974 ts_tablet_manager.cc:1950] T 31106fd8f70345bd842ebe7913e38dbc P 716583253ac44e71b7c36c57e3720a0c: Deleting consensus metadata
I20251026 15:08:38.562871 23976 ts_tablet_manager.cc:1929] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:08:38.562917 23976 log.cc:1199] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/31106fd8f70345bd842ebe7913e38dbc
I20251026 15:08:38.563109 23976 ts_tablet_manager.cc:1950] T 31106fd8f70345bd842ebe7913e38dbc P efa6dfe044714b89b53be407cf256c80: Deleting consensus metadata
I20251026 15:08:38.563114 23421 catalog_manager.cc:4985] TS 716583253ac44e71b7c36c57e3720a0c (127.14.212.66:35369): tablet 31106fd8f70345bd842ebe7913e38dbc (table test-table [id=67d90592b26747a49ae106e15b603379]) successfully deleted
I20251026 15:08:38.563422 23422 catalog_manager.cc:4985] TS efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769): tablet 31106fd8f70345bd842ebe7913e38dbc (table test-table [id=67d90592b26747a49ae106e15b603379]) successfully deleted
I20251026 15:08:38.564177 23975 ts_tablet_manager.cc:1929] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:08:38.564227 23975 log.cc:1199] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/wal/wals/31106fd8f70345bd842ebe7913e38dbc
I20251026 15:08:38.564432 23975 ts_tablet_manager.cc:1950] T 31106fd8f70345bd842ebe7913e38dbc P e9473e4ac53945b2ae886fd889a8cef0: Deleting consensus metadata
I20251026 15:08:38.564936 23420 catalog_manager.cc:4985] TS e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571): tablet 31106fd8f70345bd842ebe7913e38dbc (table test-table [id=67d90592b26747a49ae106e15b603379]) successfully deleted
Oct 26 15:08:38 dist-test-slave-8pgz krb5kdc[23387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 26 15:08:38 dist-test-slave-8pgz krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:38 dist-test-slave-8pgz krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
W20251026 15:08:38.582657 23504 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:38.576566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.14.212.126:33841)
1026 15:08:38.576636 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.576638 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.576771 (+ 133us) server_negotiation.cc:408] Connection header received
1026 15:08:38.576857 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.576858 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.576867 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.576918 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:38.577901 (+ 983us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.578439 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.579278 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.579425 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.582580 (+ 3155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37530: BlockingRecv error: recv got EOF from 127.0.0.1:37530 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Oct 26 15:08:38 dist-test-slave-8pgz krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM
I20251026 15:08:38.591105 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.583719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.14.212.126:33841)
1026 15:08:38.583760 (+ 41us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.583762 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.583772 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:38.583827 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.583828 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.583836 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.583887 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:38.584630 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.585136 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.585860 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.585994 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.588438 (+ 2444us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:38.588442 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:38.588443 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:38.588450 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:38.589919 (+ 1469us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.590297 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.590298 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.590298 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.590315 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:38.590609 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:38.590610 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:38.590610 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:38.590712 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:38.590794 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.590942 (+ 148us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.591012 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251026 15:08:38.592630 23433 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:08:38.592780 23433 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:08:38.594380 23716 tablet_service.cc:1505] Processing CreateTablet for tablet 87e528e0a60d40b7ba7542cf35914071 (DEFAULT_TABLE table=test-table [id=16ab913339eb4e98be65c9a9abc65b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:38.594380 23574 tablet_service.cc:1505] Processing CreateTablet for tablet 87e528e0a60d40b7ba7542cf35914071 (DEFAULT_TABLE table=test-table [id=16ab913339eb4e98be65c9a9abc65b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:38.594491 23858 tablet_service.cc:1505] Processing CreateTablet for tablet 87e528e0a60d40b7ba7542cf35914071 (DEFAULT_TABLE table=test-table [id=16ab913339eb4e98be65c9a9abc65b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:38.594540 23574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87e528e0a60d40b7ba7542cf35914071. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.594540 23716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87e528e0a60d40b7ba7542cf35914071. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.594610 23858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87e528e0a60d40b7ba7542cf35914071. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.595861 23953 tablet_bootstrap.cc:492] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c: Bootstrap starting.
I20251026 15:08:38.595873 23952 tablet_bootstrap.cc:492] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80: Bootstrap starting.
I20251026 15:08:38.596345 23953 tablet_bootstrap.cc:654] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.596349 23952 tablet_bootstrap.cc:654] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.596865 23956 tablet_bootstrap.cc:492] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0: Bootstrap starting.
I20251026 15:08:38.597314 23956 tablet_bootstrap.cc:654] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.597349 23953 tablet_bootstrap.cc:492] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c: No bootstrap required, opened a new log
I20251026 15:08:38.597393 23953 ts_tablet_manager.cc:1403] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 15:08:38.597594 23953 raft_consensus.cc:359] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.597679 23953 raft_consensus.cc:385] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.597707 23953 raft_consensus.cc:740] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716583253ac44e71b7c36c57e3720a0c, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.597754 23953 consensus_queue.cc:260] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.597879 23953 ts_tablet_manager.cc:1434] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 15:08:38.598225 23956 tablet_bootstrap.cc:492] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0: No bootstrap required, opened a new log
I20251026 15:08:38.598237 23952 tablet_bootstrap.cc:492] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80: No bootstrap required, opened a new log
I20251026 15:08:38.598274 23956 ts_tablet_manager.cc:1403] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:38.598281 23952 ts_tablet_manager.cc:1403] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 15:08:38.598476 23956 raft_consensus.cc:359] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.598536 23956 raft_consensus.cc:385] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.598554 23956 raft_consensus.cc:740] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9473e4ac53945b2ae886fd889a8cef0, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.598510 23952 raft_consensus.cc:359] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.598578 23952 raft_consensus.cc:385] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.598604 23952 raft_consensus.cc:740] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa6dfe044714b89b53be407cf256c80, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.598593 23956 consensus_queue.cc:260] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.598655 23952 consensus_queue.cc:260] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.598677 23956 ts_tablet_manager.cc:1434] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:38.598753 23952 ts_tablet_manager.cc:1434] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:38.697538 23995 raft_consensus.cc:493] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:08:38.697661 23995 raft_consensus.cc:515] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.697844 23995 leader_election.cc:290] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571), efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769)
I20251026 15:08:38.698096 23877 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87e528e0a60d40b7ba7542cf35914071" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa6dfe044714b89b53be407cf256c80" is_pre_election: true
I20251026 15:08:38.698132 23593 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87e528e0a60d40b7ba7542cf35914071" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9473e4ac53945b2ae886fd889a8cef0" is_pre_election: true
I20251026 15:08:38.698204 23877 raft_consensus.cc:2468] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 0.
I20251026 15:08:38.698225 23593 raft_consensus.cc:2468] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 0.
I20251026 15:08:38.698431 23670 leader_election.cc:304] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716583253ac44e71b7c36c57e3720a0c, efa6dfe044714b89b53be407cf256c80; no voters:
I20251026 15:08:38.698537 23995 raft_consensus.cc:2804] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:08:38.698597 23995 raft_consensus.cc:493] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:08:38.698628 23995 raft_consensus.cc:3060] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.699600 23995 raft_consensus.cc:515] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.699761 23995 leader_election.cc:290] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 election: Requested vote from peers e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571), efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769)
I20251026 15:08:38.700004 23877 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87e528e0a60d40b7ba7542cf35914071" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa6dfe044714b89b53be407cf256c80"
I20251026 15:08:38.700007 23593 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87e528e0a60d40b7ba7542cf35914071" candidate_uuid: "716583253ac44e71b7c36c57e3720a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9473e4ac53945b2ae886fd889a8cef0"
I20251026 15:08:38.700093 23593 raft_consensus.cc:3060] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.700090 23877 raft_consensus.cc:3060] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.700721 23877 raft_consensus.cc:2468] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 1.
I20251026 15:08:38.700922 23670 leader_election.cc:304] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716583253ac44e71b7c36c57e3720a0c, efa6dfe044714b89b53be407cf256c80; no voters:
I20251026 15:08:38.700950 23593 raft_consensus.cc:2468] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716583253ac44e71b7c36c57e3720a0c in term 1.
I20251026 15:08:38.701009 23995 raft_consensus.cc:2804] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:38.701052 23995 raft_consensus.cc:697] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [term 1 LEADER]: Becoming Leader. State: Replica: 716583253ac44e71b7c36c57e3720a0c, State: Running, Role: LEADER
I20251026 15:08:38.701144 23995 consensus_queue.cc:237] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.701879 23434 catalog_manager.cc:5649] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 716583253ac44e71b7c36c57e3720a0c (127.14.212.66). New cstate: current_term: 1 leader_uuid: "716583253ac44e71b7c36c57e3720a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } health_report { overall_health: UNKNOWN } } }
I20251026 15:08:38.723742 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.721358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.14.212.66:35369)
1026 15:08:38.721421 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.721423 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.721433 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:38.721463 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.721464 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.721476 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.721563 (+ 87us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:08:38.721779 (+ 216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.722497 (+ 718us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.723308 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.723448 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.723469 (+ 21us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:08:38.723493 (+ 24us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:08:38.723548 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.723648 (+ 100us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.723670 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251026 15:08:38.724553 23877 raft_consensus.cc:1275] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Refusing update from remote peer 716583253ac44e71b7c36c57e3720a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:38.724565 23593 raft_consensus.cc:1275] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Refusing update from remote peer 716583253ac44e71b7c36c57e3720a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:38.724812 23995 consensus_queue.cc:1048] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:38.724905 23958 consensus_queue.cc:1048] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:38.727425 23434 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37542:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:38.727517 23434 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37542:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:38.728346 23434 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a: Sending DeleteTablet for 3 replicas of tablet 87e528e0a60d40b7ba7542cf35914071
I20251026 15:08:38.728577 23716 tablet_service.cc:1552] Processing DeleteTablet for tablet 87e528e0a60d40b7ba7542cf35914071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:38 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:41574
I20251026 15:08:38.728703 23574 tablet_service.cc:1552] Processing DeleteTablet for tablet 87e528e0a60d40b7ba7542cf35914071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:38 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:59958
I20251026 15:08:38.728708 23858 tablet_service.cc:1552] Processing DeleteTablet for tablet 87e528e0a60d40b7ba7542cf35914071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:38 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:34598
I20251026 15:08:38.728724 23857 tablet_service.cc:1552] Processing DeleteTablet for tablet 87e528e0a60d40b7ba7542cf35914071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:38 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:34598
I20251026 15:08:38.728804 23974 tablet_replica.cc:333] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c: stopping tablet replica
I20251026 15:08:38.728834 23975 tablet_replica.cc:333] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0: stopping tablet replica
I20251026 15:08:38.728853 23974 raft_consensus.cc:2243] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:08:38.728883 23975 raft_consensus.cc:2243] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:38.728926 23974 raft_consensus.cc:2272] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:38.728935 23975 raft_consensus.cc:2272] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:38.729001 23976 tablet_replica.cc:333] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80: stopping tablet replica
I20251026 15:08:38.729056 23976 raft_consensus.cc:2243] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:38.729106 23976 raft_consensus.cc:2272] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:38.729439 23975 ts_tablet_manager.cc:1916] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:38.729528 23976 ts_tablet_manager.cc:1916] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:38.729563 23974 ts_tablet_manager.cc:1916] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:38.730923 23434 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 15:08:38.731027 23976 ts_tablet_manager.cc:1929] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:08:38.731071 23976 log.cc:1199] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/87e528e0a60d40b7ba7542cf35914071
I20251026 15:08:38.731078 23974 ts_tablet_manager.cc:1929] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20251026 15:08:38.731103 23434 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:08:38.731129 23974 log.cc:1199] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/87e528e0a60d40b7ba7542cf35914071
I20251026 15:08:38.731299 23976 ts_tablet_manager.cc:1950] T 87e528e0a60d40b7ba7542cf35914071 P efa6dfe044714b89b53be407cf256c80: Deleting consensus metadata
I20251026 15:08:38.731343 23974 ts_tablet_manager.cc:1950] T 87e528e0a60d40b7ba7542cf35914071 P 716583253ac44e71b7c36c57e3720a0c: Deleting consensus metadata
I20251026 15:08:38.731524 23975 ts_tablet_manager.cc:1929] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:08:38.731575 23975 log.cc:1199] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-0/wal/wals/87e528e0a60d40b7ba7542cf35914071
I20251026 15:08:38.731770 23975 ts_tablet_manager.cc:1950] T 87e528e0a60d40b7ba7542cf35914071 P e9473e4ac53945b2ae886fd889a8cef0: Deleting consensus metadata
I20251026 15:08:38.731849 23421 catalog_manager.cc:4985] TS 716583253ac44e71b7c36c57e3720a0c (127.14.212.66:35369): tablet 87e528e0a60d40b7ba7542cf35914071 (table test-table [id=16ab913339eb4e98be65c9a9abc65b5b]) successfully deleted
I20251026 15:08:38.732064 23420 catalog_manager.cc:4985] TS e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571): tablet 87e528e0a60d40b7ba7542cf35914071 (table test-table [id=16ab913339eb4e98be65c9a9abc65b5b]) successfully deleted
I20251026 15:08:38.732417 23422 catalog_manager.cc:4985] TS efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769): tablet 87e528e0a60d40b7ba7542cf35914071 (table test-table [id=16ab913339eb4e98be65c9a9abc65b5b]) successfully deleted
W20251026 15:08:38.732517 23422 catalog_manager.cc:4952] TS efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769): delete failed for tablet 87e528e0a60d40b7ba7542cf35914071 because the tablet was not found. No further retry: Not found: Tablet not found: 87e528e0a60d40b7ba7542cf35914071
I20251026 15:08:38.732609 23716 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf2615bbfa649e181c27b56e93e169b (DEFAULT_TABLE table=test-table [id=473404dd24f14dcc96c0720dee2bd323]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:38.732638 23574 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf2615bbfa649e181c27b56e93e169b (DEFAULT_TABLE table=test-table [id=473404dd24f14dcc96c0720dee2bd323]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:38.732705 23716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf2615bbfa649e181c27b56e93e169b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.732760 23574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf2615bbfa649e181c27b56e93e169b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.733588 23956 tablet_bootstrap.cc:492] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0: Bootstrap starting.
I20251026 15:08:38.733618 23857 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf2615bbfa649e181c27b56e93e169b (DEFAULT_TABLE table=test-table [id=473404dd24f14dcc96c0720dee2bd323]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:08:38.733707 23857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf2615bbfa649e181c27b56e93e169b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:38.733837 23953 tablet_bootstrap.cc:492] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c: Bootstrap starting.
I20251026 15:08:38.734165 23956 tablet_bootstrap.cc:654] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.734257 23953 tablet_bootstrap.cc:654] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.735164 23953 tablet_bootstrap.cc:492] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c: No bootstrap required, opened a new log
I20251026 15:08:38.735222 23953 ts_tablet_manager.cc:1403] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:38.735383 23953 raft_consensus.cc:359] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.735464 23953 raft_consensus.cc:385] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.735486 23953 raft_consensus.cc:740] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716583253ac44e71b7c36c57e3720a0c, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.735632 23953 consensus_queue.cc:260] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.735759 23953 ts_tablet_manager.cc:1434] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 15:08:38.735857 23952 tablet_bootstrap.cc:492] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80: Bootstrap starting.
I20251026 15:08:38.736352 23952 tablet_bootstrap.cc:654] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:38.736893 23956 tablet_bootstrap.cc:492] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0: No bootstrap required, opened a new log
I20251026 15:08:38.736936 23956 ts_tablet_manager.cc:1403] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251026 15:08:38.737071 23956 raft_consensus.cc:359] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.737131 23956 raft_consensus.cc:385] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.737145 23956 raft_consensus.cc:740] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9473e4ac53945b2ae886fd889a8cef0, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.737183 23956 consensus_queue.cc:260] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.737324 23956 ts_tablet_manager.cc:1434] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:38.737704 23952 tablet_bootstrap.cc:492] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80: No bootstrap required, opened a new log
I20251026 15:08:38.737756 23952 ts_tablet_manager.cc:1403] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:08:38.737908 23952 raft_consensus.cc:359] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.737972 23952 raft_consensus.cc:385] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:38.737998 23952 raft_consensus.cc:740] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa6dfe044714b89b53be407cf256c80, State: Initialized, Role: FOLLOWER
I20251026 15:08:38.738045 23952 consensus_queue.cc:260] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.738149 23952 ts_tablet_manager.cc:1434] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:38.965659 23957 raft_consensus.cc:493] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:08:38.965764 23957 raft_consensus.cc:515] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.966106 23957 leader_election.cc:290] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571), 716583253ac44e71b7c36c57e3720a0c (127.14.212.66:35369)
I20251026 15:08:38.968540 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.966336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:35369 (local address 127.14.212.67:33705)
1026 15:08:38.966587 (+ 251us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.966599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.966666 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.966810 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.966811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.966813 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:38.966975 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.966976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.967654 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.967655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.968257 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.968259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.968332 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.968339 (+ 7us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.968372 (+ 33us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.968414 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":14,"thread_start_us":80,"threads_started":1}
I20251026 15:08:38.968545 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.966336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:41571 (local address 127.14.212.67:56041)
1026 15:08:38.966517 (+ 181us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:38.966532 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:38.966579 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:38.966825 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:38.966826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:38.966828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:08:38.966993 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.966995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.967697 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.967698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:38.968303 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:38.968305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.968378 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.968382 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:08:38.968416 (+ 34us) client_negotiation.cc:240] Negotiation successful
1026 15:08:38.968455 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
I20251026 15:08:38.968948 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.966462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:33705 (local address 127.14.212.66:35369)
1026 15:08:38.966526 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.966529 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.966675 (+ 146us) server_negotiation.cc:408] Connection header received
1026 15:08:38.966727 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.966728 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.966736 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.966778 (+ 42us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:38.967058 (+ 280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.967555 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.968354 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.968828 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.968839 (+ 11us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.968865 (+ 26us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.968900 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251026 15:08:38.969049 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:38.966422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:56041 (local address 127.14.212.65:41571)
1026 15:08:38.966552 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:38.966554 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:38.966569 (+ 15us) server_negotiation.cc:408] Connection header received
1026 15:08:38.966661 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:38.966662 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:38.966673 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:38.966733 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:08:38.967104 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.967588 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:38.968415 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:38.968912 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:38.968920 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:38.968944 (+ 24us) server_negotiation.cc:300] Negotiation successful
1026 15:08:38.968975 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20251026 15:08:38.969180 23736 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf2615bbfa649e181c27b56e93e169b" candidate_uuid: "efa6dfe044714b89b53be407cf256c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716583253ac44e71b7c36c57e3720a0c" is_pre_election: true
I20251026 15:08:38.969260 23593 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf2615bbfa649e181c27b56e93e169b" candidate_uuid: "efa6dfe044714b89b53be407cf256c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9473e4ac53945b2ae886fd889a8cef0" is_pre_election: true
I20251026 15:08:38.969310 23736 raft_consensus.cc:2468] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa6dfe044714b89b53be407cf256c80 in term 0.
I20251026 15:08:38.969360 23593 raft_consensus.cc:2468] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa6dfe044714b89b53be407cf256c80 in term 0.
I20251026 15:08:38.969487 23811 leader_election.cc:304] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 716583253ac44e71b7c36c57e3720a0c, efa6dfe044714b89b53be407cf256c80; no voters:
I20251026 15:08:38.969638 23957 raft_consensus.cc:2804] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:08:38.969687 23957 raft_consensus.cc:493] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:08:38.969714 23957 raft_consensus.cc:3060] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.970386 23957 raft_consensus.cc:515] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.970561 23957 leader_election.cc:290] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [CANDIDATE]: Term 1 election: Requested vote from peers e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571), 716583253ac44e71b7c36c57e3720a0c (127.14.212.66:35369)
I20251026 15:08:38.970727 23736 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf2615bbfa649e181c27b56e93e169b" candidate_uuid: "efa6dfe044714b89b53be407cf256c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716583253ac44e71b7c36c57e3720a0c"
I20251026 15:08:38.970736 23593 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf2615bbfa649e181c27b56e93e169b" candidate_uuid: "efa6dfe044714b89b53be407cf256c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9473e4ac53945b2ae886fd889a8cef0"
I20251026 15:08:38.970798 23736 raft_consensus.cc:3060] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.970800 23593 raft_consensus.cc:3060] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:38.971380 23593 raft_consensus.cc:2468] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa6dfe044714b89b53be407cf256c80 in term 1.
I20251026 15:08:38.971545 23736 raft_consensus.cc:2468] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa6dfe044714b89b53be407cf256c80 in term 1.
I20251026 15:08:38.971589 23810 leader_election.cc:304] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9473e4ac53945b2ae886fd889a8cef0, efa6dfe044714b89b53be407cf256c80; no voters:
I20251026 15:08:38.971688 23957 raft_consensus.cc:2804] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:38.971791 23957 raft_consensus.cc:697] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [term 1 LEADER]: Becoming Leader. State: Replica: efa6dfe044714b89b53be407cf256c80, State: Running, Role: LEADER
I20251026 15:08:38.971850 23957 consensus_queue.cc:237] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } }
I20251026 15:08:38.972512 23434 catalog_manager.cc:5649] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 reported cstate change: term changed from 0 to 1, leader changed from <none> to efa6dfe044714b89b53be407cf256c80 (127.14.212.67). New cstate: current_term: 1 leader_uuid: "efa6dfe044714b89b53be407cf256c80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efa6dfe044714b89b53be407cf256c80" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 42769 } health_report { overall_health: HEALTHY } } }
I20251026 15:08:39.026823 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:39.024707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.14.212.67:42769)
1026 15:08:39.024842 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:39.024844 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:39.024850 (+ 6us) server_negotiation.cc:408] Connection header received
1026 15:08:39.024885 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:39.024886 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:39.024904 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:39.024959 (+ 55us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:08:39.025248 (+ 289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:39.025751 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:39.026283 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:39.026442 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:39.026482 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:08:39.026600 (+ 118us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:08:39.026664 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:39.026692 (+ 28us) server_negotiation.cc:300] Negotiation successful
1026 15:08:39.026717 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20251026 15:08:39.031205 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:39.029131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:59365 (local address 127.14.212.66:35369)
1026 15:08:39.029182 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:39.029185 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:39.029287 (+ 102us) server_negotiation.cc:408] Connection header received
1026 15:08:39.029348 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:39.029349 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:39.029382 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:39.029436 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:08:39.029722 (+ 286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:39.030182 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:39.030785 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:39.030935 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:39.030954 (+ 19us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:08:39.030976 (+ 22us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:08:39.031027 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:39.031103 (+ 76us) server_negotiation.cc:300] Negotiation successful
1026 15:08:39.031126 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251026 15:08:39.031227 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:39.029030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:35369 (local address 127.14.212.67:59365)
1026 15:08:39.029214 (+ 184us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:39.029227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:39.029289 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:39.029445 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:39.029446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:39.029447 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 15:08:39.029649 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:39.029650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:39.030261 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:39.030262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:39.030702 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:39.030703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:39.030773 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:39.030794 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 15:08:39.031039 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 15:08:39.031042 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:08:39.031093 (+ 51us) client_negotiation.cc:240] Negotiation successful
1026 15:08:39.031151 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20251026 15:08:39.033478 23736 raft_consensus.cc:1275] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Refusing update from remote peer efa6dfe044714b89b53be407cf256c80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:39.033499 23593 raft_consensus.cc:1275] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Refusing update from remote peer efa6dfe044714b89b53be407cf256c80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:08:39.033777 24026 consensus_queue.cc:1048] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "716583253ac44e71b7c36c57e3720a0c" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:39.033898 23957 consensus_queue.cc:1048] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9473e4ac53945b2ae886fd889a8cef0" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:08:39.041119 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:39.038913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:46551 (local address 127.14.212.67:42769)
1026 15:08:39.038968 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:39.038970 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:39.038978 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:08:39.039002 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:39.039002 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:39.039016 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:39.039140 (+ 124us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:08:39.039391 (+ 251us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:39.039882 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:39.040596 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:39.040801 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:39.040826 (+ 25us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:08:39.040856 (+ 30us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:08:39.040937 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:39.041011 (+ 74us) server_negotiation.cc:300] Negotiation successful
1026 15:08:39.041036 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251026 15:08:39.041129 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:39.038746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:42769 (local address 127.14.212.66:46551)
1026 15:08:39.038787 (+ 41us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:39.038799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:39.038836 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:39.039163 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:39.039164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:39.039166 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
1026 15:08:39.039326 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:39.039328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:39.039953 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:39.039954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:39.040474 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:39.040477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:39.040572 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:39.040580 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1026 15:08:39.040954 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1026 15:08:39.040956 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:08:39.041003 (+ 47us) client_negotiation.cc:240] Negotiation successful
1026 15:08:39.041065 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20251026 15:08:39.047610 23433 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37542:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:39.047725 23433 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37542:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:08:39.048519 23433 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 3ec2c8f4473247ceaf108e8dc22c800a: Sending DeleteTablet for 3 replicas of tablet 3cf2615bbfa649e181c27b56e93e169b
I20251026 15:08:39.048780 23574 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf2615bbfa649e181c27b56e93e169b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:39 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:59958
I20251026 15:08:39.048877 23975 tablet_replica.cc:333] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0: stopping tablet replica
I20251026 15:08:39.048890 23857 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf2615bbfa649e181c27b56e93e169b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:39 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:34598
I20251026 15:08:39.048923 23975 raft_consensus.cc:2243] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:39.048954 23975 raft_consensus.cc:2272] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:39.049005 23976 tablet_replica.cc:333] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80: stopping tablet replica
I20251026 15:08:39.049054 23976 raft_consensus.cc:2243] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:08:39.049144 23976 raft_consensus.cc:2272] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:39.049409 23716 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf2615bbfa649e181c27b56e93e169b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:08:39 UTC) from {username='oryx', principal='oryx/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:41574
I20251026 15:08:39.049439 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 23509
I20251026 15:08:39.049507 23974 tablet_replica.cc:333] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c: stopping tablet replica
I20251026 15:08:39.049548 23974 raft_consensus.cc:2243] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:08:39.049587 23974 raft_consensus.cc:2272] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:08:39.049592 23975 ts_tablet_manager.cc:1916] T 3cf2615bbfa649e181c27b56e93e169b P e9473e4ac53945b2ae886fd889a8cef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:39.053362 23974 ts_tablet_manager.cc:1916] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:39.054510 23976 ts_tablet_manager.cc:1916] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:08:39.055025 23974 ts_tablet_manager.cc:1929] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:08:39.055092 23974 log.cc:1199] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/3cf2615bbfa649e181c27b56e93e169b
I20251026 15:08:39.055326 23974 ts_tablet_manager.cc:1950] T 3cf2615bbfa649e181c27b56e93e169b P 716583253ac44e71b7c36c57e3720a0c: Deleting consensus metadata
I20251026 15:08:39.055625 23976 ts_tablet_manager.cc:1929] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251026 15:08:39.055689 23976 log.cc:1199] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/3cf2615bbfa649e181c27b56e93e169b
I20251026 15:08:39.055773 23421 catalog_manager.cc:4985] TS 716583253ac44e71b7c36c57e3720a0c (127.14.212.66:35369): tablet 3cf2615bbfa649e181c27b56e93e169b (table test-table [id=473404dd24f14dcc96c0720dee2bd323]) successfully deleted
I20251026 15:08:39.055902 23976 ts_tablet_manager.cc:1950] T 3cf2615bbfa649e181c27b56e93e169b P efa6dfe044714b89b53be407cf256c80: Deleting consensus metadata
I20251026 15:08:39.056222 23422 catalog_manager.cc:4985] TS efa6dfe044714b89b53be407cf256c80 (127.14.212.67:42769): tablet 3cf2615bbfa649e181c27b56e93e169b (table test-table [id=473404dd24f14dcc96c0720dee2bd323]) successfully deleted
W20251026 15:08:39.059464 23420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:41571 (error 108)
I20251026 15:08:39.059818 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 23651
I20251026 15:08:39.060662 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:39.059880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:41571 (local address 127.0.0.1:59960)
1026 15:08:39.060553 (+ 673us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:39.060608 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:41571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":53,"threads_started":1}
W20251026 15:08:39.060776 23420 catalog_manager.cc:4712] TS e9473e4ac53945b2ae886fd889a8cef0 (127.14.212.65:41571): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3cf2615bbfa649e181c27b56e93e169b: Network error: Client connection negotiation failed: client connection to 127.14.212.65:41571: connect: Connection refused (error 111)
I20251026 15:08:39.067322 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 23793
I20251026 15:08:39.072460 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 23403
2025-10-26T15:08:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3619 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:08:39 dist-test-slave-8pgz krb5kdc[24040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:08:39 dist-test-slave-8pgz krb5kdc[24040](info): set up 2 sockets
Oct 26 15:08:39 dist-test-slave-8pgz krb5kdc[24040](info): commencing operation
krb5kdc: starting...
W20251026 15:08:41.060542 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.966s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:08:41 dist-test-slave-8pgz krb5kdc[24040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:08:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:41.199290 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:34179
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:34179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:41.281803 24056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:41.282003 24056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:41.282034 24056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:41.283526 24056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:41.283577 24056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:41.283594 24056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:41.283610 24056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:41.283624 24056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:08:41.285219 24056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:34179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:34179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:41.285418 24056 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:41.285571 24056 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:41.288422 24062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:41.288529 24056 server_base.cc:1047] running on GCE node
W20251026 15:08:41.288424 24064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:41.288424 24061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:41.288869 24056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:41.289105 24056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:41.290293 24056 hybrid_clock.cc:648] HybridClock initialized: now 1761491321290258 us; error 32 us; skew 500 ppm
W20251026 15:08:41.292104 24056 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-10-26T15:08:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2223 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:08:41 dist-test-slave-8pgz krb5kdc[24069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:08:41 dist-test-slave-8pgz krb5kdc[24069](info): set up 2 sockets
Oct 26 15:08:41 dist-test-slave-8pgz krb5kdc[24069](info): commencing operation
krb5kdc: starting...
W20251026 15:08:43.304507 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.987s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:08:43 dist-test-slave-8pgz krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:08:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:43.439136 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33551
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:43.519243 24085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:43.519429 24085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:43.519454 24085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:43.520828 24085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:43.520876 24085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:43.520890 24085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:43.520902 24085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:43.520913 24085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:08:43.522470 24085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:43.522648 24085 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:43.522787 24085 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:43.525417 24093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:43.525429 24091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:43.525460 24085 server_base.cc:1047] running on GCE node
W20251026 15:08:43.525429 24090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:43.525858 24085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:43.526063 24085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:43.527199 24085 hybrid_clock.cc:648] HybridClock initialized: now 1761491323527180 us; error 39 us; skew 500 ppm
Oct 26 15:08:43 dist-test-slave-8pgz krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491323, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:43.529735 24085 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:08:43.530262 24085 webserver.cc:492] Webserver started at http://127.14.212.126:36351/ using document root <none> and password file <none>
I20251026 15:08:43.530489 24085 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:43.530535 24085 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:43.530651 24085 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:43.531608 24085 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "05f4719942a44c8f84bf8c2766ae3b61"
format_stamp: "Formatted at 2025-10-26 15:08:43 on dist-test-slave-8pgz"
server_key: "3e4833add97fea9e45130a9ea9c9aad5"
server_key_iv: "0a8f6f9d44239642c3b22fb42a06e422"
server_key_version: "encryptionkey@0"
I20251026 15:08:43.531906 24085 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "05f4719942a44c8f84bf8c2766ae3b61"
format_stamp: "Formatted at 2025-10-26 15:08:43 on dist-test-slave-8pgz"
server_key: "3e4833add97fea9e45130a9ea9c9aad5"
server_key_iv: "0a8f6f9d44239642c3b22fb42a06e422"
server_key_version: "encryptionkey@0"
I20251026 15:08:43.533087 24085 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:08:43.533756 24100 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:43.533914 24085 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:08:43.533998 24085 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "05f4719942a44c8f84bf8c2766ae3b61"
format_stamp: "Formatted at 2025-10-26 15:08:43 on dist-test-slave-8pgz"
server_key: "3e4833add97fea9e45130a9ea9c9aad5"
server_key_iv: "0a8f6f9d44239642c3b22fb42a06e422"
server_key_version: "encryptionkey@0"
I20251026 15:08:43.534058 24085 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:43.544119 24085 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:43.545936 24085 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:43.546061 24085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:43.549768 24085 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:33551
I20251026 15:08:43.549818 24152 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:33551 every 8 connection(s)
I20251026 15:08:43.550114 24085 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:43.550729 24153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:43.553130 24153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61: Bootstrap starting.
I20251026 15:08:43.553661 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 24085
I20251026 15:08:43.553774 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:08:43.553782 24153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:43.553977 15185 external_mini_cluster.cc:1468] Setting key 14621987f355c0b46f3920b483e380ff
I20251026 15:08:43.554044 24153 log.cc:826] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:43.554834 24153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61: No bootstrap required, opened a new log
I20251026 15:08:43.555866 24153 raft_consensus.cc:359] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4719942a44c8f84bf8c2766ae3b61" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33551 } }
I20251026 15:08:43.555984 24153 raft_consensus.cc:385] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:43.556016 24153 raft_consensus.cc:740] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4719942a44c8f84bf8c2766ae3b61, State: Initialized, Role: FOLLOWER
I20251026 15:08:43.556114 24153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4719942a44c8f84bf8c2766ae3b61" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33551 } }
I20251026 15:08:43.556191 24153 raft_consensus.cc:399] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:43.556234 24153 raft_consensus.cc:493] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:43.556279 24153 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:43.556810 24153 raft_consensus.cc:515] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4719942a44c8f84bf8c2766ae3b61" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33551 } }
I20251026 15:08:43.556917 24153 leader_election.cc:304] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f4719942a44c8f84bf8c2766ae3b61; no voters:
I20251026 15:08:43.557058 24153 leader_election.cc:290] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:08:43.557108 24158 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:43.557264 24153 sys_catalog.cc:565] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:43.557273 24158 raft_consensus.cc:697] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 1 LEADER]: Becoming Leader. State: Replica: 05f4719942a44c8f84bf8c2766ae3b61, State: Running, Role: LEADER
I20251026 15:08:43.557706 24158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4719942a44c8f84bf8c2766ae3b61" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33551 } }
I20251026 15:08:43.558257 24160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05f4719942a44c8f84bf8c2766ae3b61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4719942a44c8f84bf8c2766ae3b61" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33551 } } }
I20251026 15:08:43.558337 24160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:43.558863 24159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05f4719942a44c8f84bf8c2766ae3b61. Latest consensus state: current_term: 1 leader_uuid: "05f4719942a44c8f84bf8c2766ae3b61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4719942a44c8f84bf8c2766ae3b61" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33551 } } }
I20251026 15:08:43.558923 24169 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:43.559356 24169 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:08:43.558934 24159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [sys.catalog]: This master's current role is: LEADER
Oct 26 15:08:43 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:43.560724 24169 catalog_manager.cc:1357] Generated new cluster ID: 9d8dc1093eaa4501a2abe5939f0ed264
I20251026 15:08:43.560781 24169 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:08:43.564388 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:43.554493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37242 (local address 127.14.212.126:33551)
1026 15:08:43.554671 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:43.554682 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:43.554703 (+ 21us) server_negotiation.cc:408] Connection header received
1026 15:08:43.555033 (+ 330us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:43.555039 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:43.555081 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:43.555209 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:43.555949 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:43.556807 (+ 858us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:43.557583 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:43.557823 (+ 240us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:43.560838 (+ 3015us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:43.560860 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:43.560860 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:43.560881 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:43.563005 (+ 2124us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:43.563376 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:43.563379 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:43.563379 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:43.563439 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:43.563590 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:43.563591 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:43.563592 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:43.563835 (+ 243us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:43.563975 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:43.564078 (+ 103us) server_negotiation.cc:300] Negotiation successful
1026 15:08:43.564213 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20251026 15:08:43.567636 24169 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:08:43.568351 24169 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:43.577239 24169 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61: Generated new TSK 0
I20251026 15:08:43.577440 24169 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:43.635317 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33551
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:43.717133 24182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:43.717339 24182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:43.717371 24182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:43.718911 24182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:43.718973 24182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:43.719003 24182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:08:43.720728 24182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:43.720986 24182 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:43.721192 24182 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:43.724263 24190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:43.724351 24188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:43.724259 24187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:43.724426 24182 server_base.cc:1047] running on GCE node
I20251026 15:08:43.724684 24182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:43.724886 24182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:43.726029 24182 hybrid_clock.cc:648] HybridClock initialized: now 1761491323726017 us; error 32 us; skew 500 ppm
Oct 26 15:08:43 dist-test-slave-8pgz krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491323, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:43.728425 24182 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:08:43.728968 24182 webserver.cc:492] Webserver started at http://127.14.212.65:44271/ using document root <none> and password file <none>
I20251026 15:08:43.729161 24182 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:43.729202 24182 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:43.729319 24182 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:43.730320 24182 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "c1604b33b7e34d12bbffbb0ed7a94a22"
format_stamp: "Formatted at 2025-10-26 15:08:43 on dist-test-slave-8pgz"
server_key: "271fee70cbfd988174d7337f7d3ba4f7"
server_key_iv: "487c03381dbf9756c47adaa4d5b24712"
server_key_version: "encryptionkey@0"
I20251026 15:08:43.730679 24182 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "c1604b33b7e34d12bbffbb0ed7a94a22"
format_stamp: "Formatted at 2025-10-26 15:08:43 on dist-test-slave-8pgz"
server_key: "271fee70cbfd988174d7337f7d3ba4f7"
server_key_iv: "487c03381dbf9756c47adaa4d5b24712"
server_key_version: "encryptionkey@0"
I20251026 15:08:43.731941 24182 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:43.732788 24197 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:43.732968 24182 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:43.733105 24182 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "c1604b33b7e34d12bbffbb0ed7a94a22"
format_stamp: "Formatted at 2025-10-26 15:08:43 on dist-test-slave-8pgz"
server_key: "271fee70cbfd988174d7337f7d3ba4f7"
server_key_iv: "487c03381dbf9756c47adaa4d5b24712"
server_key_version: "encryptionkey@0"
I20251026 15:08:43.733173 24182 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:43.753679 24182 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:43.756150 24182 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:43.756311 24182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:43.756553 24182 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:43.756887 24182 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:43.756924 24182 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:43.756959 24182 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:43.756987 24182 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:43.763008 24182 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:39711
I20251026 15:08:43.763111 24310 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:39711 every 8 connection(s)
I20251026 15:08:43.763391 24182 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
Oct 26 15:08:43 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491323, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:43.770924 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 24182
I20251026 15:08:43.771054 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:08:43.771317 15185 external_mini_cluster.cc:1468] Setting key 0d35c45ae1d7b2ab5efd195557118edd
I20251026 15:08:43.772114 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:43.764161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:41025 (local address 127.14.212.126:33551)
1026 15:08:43.764269 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:43.764272 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:43.764433 (+ 161us) server_negotiation.cc:408] Connection header received
1026 15:08:43.764933 (+ 500us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:43.764935 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:43.764947 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:43.765009 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:43.765810 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:43.766310 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:43.767362 (+ 1052us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:43.767550 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:43.769431 (+ 1881us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:43.769436 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:43.769436 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:43.769443 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:43.770795 (+ 1352us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:43.771205 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:43.771208 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:43.771208 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:43.771247 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:43.771446 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:43.771447 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:43.771447 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:43.771567 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:43.771636 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:43.771939 (+ 303us) server_negotiation.cc:300] Negotiation successful
1026 15:08:43.772048 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251026 15:08:43.772281 24313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:43.764090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:41025)
1026 15:08:43.764289 (+ 199us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:43.764324 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:43.764830 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:43.765081 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:43.765084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:43.765412 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:43.765695 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:43.765699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:43.766424 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:43.766425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:43.767264 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:43.767266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:43.767403 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:43.767998 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:43.768017 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:43.769342 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:43.770872 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:43.770878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:43.770881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:43.771112 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:43.771322 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:43.771323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:43.771323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:43.771381 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:43.771648 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:43.771654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:43.771796 (+ 142us) client_negotiation.cc:769] Sending connection context
1026 15:08:43.771903 (+ 107us) client_negotiation.cc:240] Negotiation successful
1026 15:08:43.772069 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
I20251026 15:08:43.772830 24311 heartbeater.cc:344] Connected to a master server at 127.14.212.126:33551
I20251026 15:08:43.772930 24311 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:43.773128 24311 heartbeater.cc:507] Master 127.14.212.126:33551 requested a full tablet report, sending...
I20251026 15:08:43.773680 24117 ts_manager.cc:194] Registered new tserver with Master: c1604b33b7e34d12bbffbb0ed7a94a22 (127.14.212.65:39711)
I20251026 15:08:43.774713 24117 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:41025
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:43.821089 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33551
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:43.901297 24318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:43.901499 24318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:43.901525 24318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:43.903039 24318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:43.903095 24318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:43.903131 24318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:08:43.904707 24318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:43.904960 24318 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:43.905158 24318 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:08:43.907887 24318 server_base.cc:1047] running on GCE node
W20251026 15:08:43.907855 24323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:43.907876 24326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:43.907859 24324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:43.908309 24318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:43.908545 24318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:43.909680 24318 hybrid_clock.cc:648] HybridClock initialized: now 1761491323909670 us; error 35 us; skew 500 ppm
Oct 26 15:08:43 dist-test-slave-8pgz krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491323, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:43.912254 24318 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:08:43.912786 24318 webserver.cc:492] Webserver started at http://127.14.212.66:34993/ using document root <none> and password file <none>
I20251026 15:08:43.912983 24318 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:43.913022 24318 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:43.913146 24318 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:43.914113 24318 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "8eedf209ac3e4314ab33cf804684d090"
format_stamp: "Formatted at 2025-10-26 15:08:43 on dist-test-slave-8pgz"
server_key: "0b7e521af152d34a643db9a0f99f072b"
server_key_iv: "5f8423835bea147f2d7f4607b477b248"
server_key_version: "encryptionkey@0"
I20251026 15:08:43.914487 24318 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "8eedf209ac3e4314ab33cf804684d090"
format_stamp: "Formatted at 2025-10-26 15:08:43 on dist-test-slave-8pgz"
server_key: "0b7e521af152d34a643db9a0f99f072b"
server_key_iv: "5f8423835bea147f2d7f4607b477b248"
server_key_version: "encryptionkey@0"
I20251026 15:08:43.915714 24318 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:43.916420 24333 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:43.916615 24318 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:43.916689 24318 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "8eedf209ac3e4314ab33cf804684d090"
format_stamp: "Formatted at 2025-10-26 15:08:43 on dist-test-slave-8pgz"
server_key: "0b7e521af152d34a643db9a0f99f072b"
server_key_iv: "5f8423835bea147f2d7f4607b477b248"
server_key_version: "encryptionkey@0"
I20251026 15:08:43.916754 24318 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:43.935737 24318 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:43.938486 24318 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:43.938627 24318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:43.938875 24318 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:43.939266 24318 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:43.939308 24318 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:43.939342 24318 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:43.939363 24318 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:43.946872 24318 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:41273
I20251026 15:08:43.946907 24446 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:41273 every 8 connection(s)
I20251026 15:08:43.947224 24318 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
Oct 26 15:08:43 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491323, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:43.955878 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 24318
I20251026 15:08:43.955991 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:08:43.956185 15185 external_mini_cluster.cc:1468] Setting key 21547830db78f9604e17938ad3b52d01
I20251026 15:08:43.956578 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:43.948041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:50477 (local address 127.14.212.126:33551)
1026 15:08:43.948113 (+ 72us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:43.948116 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:43.948281 (+ 165us) server_negotiation.cc:408] Connection header received
1026 15:08:43.948780 (+ 499us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:43.948781 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:43.948794 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:43.948856 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:43.949635 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:43.950128 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:43.951176 (+ 1048us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:43.951374 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:43.953584 (+ 2210us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:43.953589 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:43.953589 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:43.953597 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:43.955127 (+ 1530us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:43.955544 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:43.955545 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:43.955545 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:43.955570 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:43.955754 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:43.955755 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:43.955756 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:43.955865 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:43.955927 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:43.956375 (+ 448us) server_negotiation.cc:300] Negotiation successful
1026 15:08:43.956488 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251026 15:08:43.956670 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:43.947931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:50477)
1026 15:08:43.948176 (+ 245us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:43.948203 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:43.948685 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:43.948923 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:43.948927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:43.949263 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:43.949539 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:43.949542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:43.950213 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:43.950216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:43.951071 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:43.951073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:43.951235 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:43.951886 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:43.951906 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:43.953478 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:43.955218 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:43.955226 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:43.955229 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:43.955463 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:43.955630 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:43.955631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:43.955631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:43.955698 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:43.955922 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:43.955927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:43.956168 (+ 241us) client_negotiation.cc:769] Sending connection context
1026 15:08:43.956328 (+ 160us) client_negotiation.cc:240] Negotiation successful
1026 15:08:43.956479 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":112,"threads_started":1}
I20251026 15:08:43.957170 24447 heartbeater.cc:344] Connected to a master server at 127.14.212.126:33551
I20251026 15:08:43.957273 24447 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:43.957484 24447 heartbeater.cc:507] Master 127.14.212.126:33551 requested a full tablet report, sending...
I20251026 15:08:43.957856 24117 ts_manager.cc:194] Registered new tserver with Master: 8eedf209ac3e4314ab33cf804684d090 (127.14.212.66:41273)
I20251026 15:08:43.958441 24117 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:50477
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:08:44.005120 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:33551
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:44.085863 24454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:44.086042 24454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:44.086067 24454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:44.087574 24454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:44.087625 24454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:44.087654 24454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:08:44.089236 24454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:44.089468 24454 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:44.089661 24454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:44.092509 24459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:44.092518 24462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:44.092693 24460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:44.092803 24454 server_base.cc:1047] running on GCE node
I20251026 15:08:44.092972 24454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:44.093214 24454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:44.094436 24454 hybrid_clock.cc:648] HybridClock initialized: now 1761491324094410 us; error 41 us; skew 500 ppm
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491324, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:44.096958 24454 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:08:44.097555 24454 webserver.cc:492] Webserver started at http://127.14.212.67:38243/ using document root <none> and password file <none>
I20251026 15:08:44.097745 24454 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:44.097790 24454 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:44.097915 24454 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:44.098994 24454 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "ca8990ac7bd1465080f713cd3488e060"
format_stamp: "Formatted at 2025-10-26 15:08:44 on dist-test-slave-8pgz"
server_key: "8dc7d3888c0665e3ed9e9afbfdbbaf3f"
server_key_iv: "f90876394c3d1f1124c81174fa47dd74"
server_key_version: "encryptionkey@0"
I20251026 15:08:44.099330 24454 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "ca8990ac7bd1465080f713cd3488e060"
format_stamp: "Formatted at 2025-10-26 15:08:44 on dist-test-slave-8pgz"
server_key: "8dc7d3888c0665e3ed9e9afbfdbbaf3f"
server_key_iv: "f90876394c3d1f1124c81174fa47dd74"
server_key_version: "encryptionkey@0"
I20251026 15:08:44.100652 24454 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:08:44.101428 24469 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:44.101626 24454 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:44.101707 24454 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "ca8990ac7bd1465080f713cd3488e060"
format_stamp: "Formatted at 2025-10-26 15:08:44 on dist-test-slave-8pgz"
server_key: "8dc7d3888c0665e3ed9e9afbfdbbaf3f"
server_key_iv: "f90876394c3d1f1124c81174fa47dd74"
server_key_version: "encryptionkey@0"
I20251026 15:08:44.101776 24454 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:44.108820 24454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:44.110984 24454 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:44.111183 24454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:44.111395 24454 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:44.111750 24454 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:44.111789 24454 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:44.111824 24454 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:44.111850 24454 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:44.117635 24454 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:42007
I20251026 15:08:44.117690 24582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:42007 every 8 connection(s)
I20251026 15:08:44.118013 24454 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:08:44.119436 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 24454
I20251026 15:08:44.119546 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:08:44.119719 15185 external_mini_cluster.cc:1468] Setting key a7edf9a2a62c4fc9c7b4b0d1d7918515
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491324, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:08:44.126999 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:44.118699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:38235 (local address 127.14.212.126:33551)
1026 15:08:44.118791 (+ 92us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:44.118794 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:44.118956 (+ 162us) server_negotiation.cc:408] Connection header received
1026 15:08:44.119672 (+ 716us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:44.119673 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:44.119683 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:44.119744 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:44.120684 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.121318 (+ 634us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.122320 (+ 1002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.122515 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.124487 (+ 1972us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:44.124491 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:44.124491 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:44.124502 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:44.125916 (+ 1414us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:44.126278 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:44.126279 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:44.126279 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:44.126297 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:44.126513 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:44.126514 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:44.126514 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:44.126602 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:44.126656 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:44.126870 (+ 214us) server_negotiation.cc:300] Negotiation successful
1026 15:08:44.126934 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251026 15:08:44.127115 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:44.118644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:38235)
1026 15:08:44.118837 (+ 193us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:44.118870 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:44.119495 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:44.119828 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:44.119832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:44.120226 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:44.120609 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.120613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.121425 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.121426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:44.122235 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.122237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.122372 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.123033 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:44.123050 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:44.124410 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:08:44.125992 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:44.125998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:44.126000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:44.126216 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:44.126342 (+ 126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:08:44.126343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:08:44.126343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:08:44.126453 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:08:44.126652 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:08:44.126654 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:08:44.126769 (+ 115us) client_negotiation.cc:769] Sending connection context
1026 15:08:44.126853 (+ 84us) client_negotiation.cc:240] Negotiation successful
1026 15:08:44.126958 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20251026 15:08:44.127565 24583 heartbeater.cc:344] Connected to a master server at 127.14.212.126:33551
I20251026 15:08:44.127661 24583 heartbeater.cc:461] Registering TS with master...
I20251026 15:08:44.127861 24583 heartbeater.cc:507] Master 127.14.212.126:33551 requested a full tablet report, sending...
I20251026 15:08:44.128207 24117 ts_manager.cc:194] Registered new tserver with Master: ca8990ac7bd1465080f713cd3488e060 (127.14.212.67:42007)
I20251026 15:08:44.128569 24117 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:38235
I20251026 15:08:44.131707 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/oryx_127.14.212.65.keytab.
Entry for principal oryx/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/oryx_127.14.212.65.keytab.
WARNING: no policy specified for oryx/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/oryx_127.14.212.66.keytab.
Entry for principal oryx/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/oryx_127.14.212.66.keytab.
WARNING: no policy specified for oryx/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/oryx_127.14.212.67.keytab.
Entry for principal oryx/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/oryx_127.14.212.67.keytab.
I20251026 15:08:44.204456 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 24182
I20251026 15:08:44.209106 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 24318
I20251026 15:08:44.213495 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 24454
I20251026 15:08:44.217844 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 24085
I20251026 15:08:44.222951 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33551
--webserver_interface=127.14.212.126
--webserver_port=36351
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:44.302219 24592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:44.302440 24592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:44.302479 24592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:44.303885 24592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:44.303932 24592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:44.303953 24592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:44.303975 24592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:44.303993 24592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:08:44.305593 24592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=36351
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:44.305846 24592 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:44.306023 24592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:44.308550 24597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:44.308583 24598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:44.308552 24600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:44.308637 24592 server_base.cc:1047] running on GCE node
I20251026 15:08:44.308965 24592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:44.309171 24592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:44.310308 24592 hybrid_clock.cc:648] HybridClock initialized: now 1761491324310292 us; error 32 us; skew 500 ppm
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491324, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:44.313051 24592 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:08:44.313609 24592 webserver.cc:492] Webserver started at http://127.14.212.126:36351/ using document root <none> and password file <none>
I20251026 15:08:44.313796 24592 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:44.313846 24592 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:44.315140 24592 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:44.315724 24607 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:44.315888 24592 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:08:44.315953 24592 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "05f4719942a44c8f84bf8c2766ae3b61"
format_stamp: "Formatted at 2025-10-26 15:08:43 on dist-test-slave-8pgz"
server_key: "3e4833add97fea9e45130a9ea9c9aad5"
server_key_iv: "0a8f6f9d44239642c3b22fb42a06e422"
server_key_version: "encryptionkey@0"
I20251026 15:08:44.316223 24592 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:44.326277 24592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:44.328189 24592 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:44.328302 24592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:44.332302 24592 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:33551
I20251026 15:08:44.332338 24659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:33551 every 8 connection(s)
I20251026 15:08:44.332674 24592 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:44.333809 24660 sys_catalog.cc:263] Verifying existing consensus state
I20251026 15:08:44.334365 24660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61: Bootstrap starting.
I20251026 15:08:44.337447 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 24592
I20251026 15:08:44.337718 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:39711
--local_ip_for_outbound_sockets=127.14.212.65
--tserver_master_addrs=127.14.212.126:33551
--webserver_port=44271
--webserver_interface=127.14.212.65
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.212.65
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/oryx_127.14.212.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:08:44.343834 24660 log.cc:826] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:44.345558 24660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 15:08:44.345844 24660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61: Bootstrap complete.
I20251026 15:08:44.347955 24660 raft_consensus.cc:359] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4719942a44c8f84bf8c2766ae3b61" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33551 } }
I20251026 15:08:44.348157 24660 raft_consensus.cc:740] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4719942a44c8f84bf8c2766ae3b61, State: Initialized, Role: FOLLOWER
I20251026 15:08:44.348279 24660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4719942a44c8f84bf8c2766ae3b61" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33551 } }
I20251026 15:08:44.348341 24660 raft_consensus.cc:399] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:44.348381 24660 raft_consensus.cc:493] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:44.348428 24660 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 1 FOLLOWER]: Advancing to term 2
I20251026 15:08:44.349282 24660 raft_consensus.cc:515] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4719942a44c8f84bf8c2766ae3b61" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33551 } }
I20251026 15:08:44.349376 24660 leader_election.cc:304] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f4719942a44c8f84bf8c2766ae3b61; no voters:
I20251026 15:08:44.349548 24660 leader_election.cc:290] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251026 15:08:44.349606 24664 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 15:08:44.349717 24664 raft_consensus.cc:697] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [term 2 LEADER]: Becoming Leader. State: Replica: 05f4719942a44c8f84bf8c2766ae3b61, State: Running, Role: LEADER
I20251026 15:08:44.349790 24660 sys_catalog.cc:565] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:44.349795 24664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4719942a44c8f84bf8c2766ae3b61" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33551 } }
I20251026 15:08:44.350301 24664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05f4719942a44c8f84bf8c2766ae3b61. Latest consensus state: current_term: 2 leader_uuid: "05f4719942a44c8f84bf8c2766ae3b61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4719942a44c8f84bf8c2766ae3b61" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33551 } } }
I20251026 15:08:44.350381 24664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:44.350522 24665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "05f4719942a44c8f84bf8c2766ae3b61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4719942a44c8f84bf8c2766ae3b61" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33551 } } }
I20251026 15:08:44.350584 24669 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:44.350585 24665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:44.351045 24669 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:08:44.351181 24669 catalog_manager.cc:1269] Loaded cluster ID: 9d8dc1093eaa4501a2abe5939f0ed264
I20251026 15:08:44.351217 24669 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:08:44.352571 24669 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:44.352749 24669 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 05f4719942a44c8f84bf8c2766ae3b61: Loaded TSK: 0
I20251026 15:08:44.352953 24669 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251026 15:08:44.418787 24662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:44.418972 24662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:44.419008 24662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:44.420560 24662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:44.420615 24662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:44.420660 24662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:08:44.422333 24662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/oryx_127.14.212.65.keytab
--principal=oryx/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:39711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=44271
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:44.422619 24662 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:44.422807 24662 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:44.425390 24687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:44.425400 24686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:44.425593 24662 server_base.cc:1047] running on GCE node
W20251026 15:08:44.425417 24689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:44.425799 24662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:44.425992 24662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:44.427124 24662 hybrid_clock.cc:648] HybridClock initialized: now 1761491324427100 us; error 45 us; skew 500 ppm
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491324, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:44.429502 24662 init.cc:377] Logged in from keytab as oryx/127.14.212.65@KRBTEST.COM (short username oryx)
I20251026 15:08:44.429984 24662 webserver.cc:492] Webserver started at http://127.14.212.65:44271/ using document root <none> and password file <none>
I20251026 15:08:44.430186 24662 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:44.430249 24662 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:44.431622 24662 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:44.432171 24696 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:44.432338 24662 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:08:44.432413 24662 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "c1604b33b7e34d12bbffbb0ed7a94a22"
format_stamp: "Formatted at 2025-10-26 15:08:43 on dist-test-slave-8pgz"
server_key: "271fee70cbfd988174d7337f7d3ba4f7"
server_key_iv: "487c03381dbf9756c47adaa4d5b24712"
server_key_version: "encryptionkey@0"
I20251026 15:08:44.432679 24662 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:44.444931 24662 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:44.447114 24662 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:44.447266 24662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:44.447471 24662 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:44.447762 24662 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:44.447793 24662 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:44.447827 24662 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:44.447860 24662 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:44.453946 24662 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:39711
I20251026 15:08:44.454002 24809 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:39711 every 8 connection(s)
I20251026 15:08:44.454315 24662 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:44.462188 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 24662
I20251026 15:08:44.462553 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:41273
--local_ip_for_outbound_sockets=127.14.212.66
--tserver_master_addrs=127.14.212.126:33551
--webserver_port=34993
--webserver_interface=127.14.212.66
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.212.66
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/oryx_127.14.212.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:08:44.462687 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:44.455033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:33871)
1026 15:08:44.455179 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:44.455214 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:44.455838 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:44.456220 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:44.456225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:44.456670 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:44.457073 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.457078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.458075 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.458076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:44.459274 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.459278 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.459439 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.460006 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:44.460030 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:44.462502 (+ 2472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
W20251026 15:08:44.462922 24810 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:33551 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.212.126:33551: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
W20251026 15:08:44.463528 24813 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:44.455162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:33871 (local address 127.14.212.126:33551)
1026 15:08:44.455301 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:44.455310 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:44.455330 (+ 20us) server_negotiation.cc:408] Connection header received
1026 15:08:44.456013 (+ 683us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:44.456017 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:44.456054 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:44.456152 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:44.457184 (+ 1032us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.457975 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.459390 (+ 1415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.459566 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.463361 (+ 3795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:33871: BlockingRecv error: recv got EOF from 127.14.212.65:33871 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:44.467583 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:44.463180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:54315)
1026 15:08:44.463270 (+ 90us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:44.463282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:44.463318 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:44.464087 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:44.464089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:44.464214 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:44.464355 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.464356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.464956 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.464957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:44.465305 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.465307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.465373 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.465880 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:44.465885 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:44.467502 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20251026 15:08:44.467983 24813 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:44.463911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:54315 (local address 127.14.212.126:33551)
1026 15:08:44.463963 (+ 52us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:44.463966 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:44.463975 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:08:44.464000 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:44.464002 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:44.464012 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:44.464069 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:44.464432 (+ 363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.464885 (+ 453us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.465376 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.465924 (+ 548us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.467603 (+ 1679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:54315: BlockingRecv error: recv got EOF from 127.14.212.65:54315 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:44.471779 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:44.467903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:41275)
1026 15:08:44.468219 (+ 316us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:44.468229 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:44.468283 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:44.468415 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:44.468415 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:44.468542 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:44.468649 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.468651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.469259 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.469260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:44.469614 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.469616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.469683 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.470214 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:44.470218 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:44.471712 (+ 1494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302}
W20251026 15:08:44.471904 24810 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251026 15:08:44.472045 24813 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:44.467952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:41275 (local address 127.14.212.126:33551)
1026 15:08:44.468190 (+ 238us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:44.468192 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:44.468279 (+ 87us) server_negotiation.cc:408] Connection header received
1026 15:08:44.468344 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:44.468345 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:44.468358 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:44.468407 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:44.468717 (+ 310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.469163 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.469687 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.469891 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.471802 (+ 1911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:41275: BlockingRecv error: recv got EOF from 127.14.212.65:41275 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":99,"threads_started":1}
W20251026 15:08:44.566061 24814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:44.566246 24814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:44.566269 24814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:44.567728 24814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:44.567780 24814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:44.567806 24814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:08:44.569320 24814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/oryx_127.14.212.66.keytab
--principal=oryx/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:41273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=34993
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:44.569538 24814 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:44.569734 24814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:44.572365 24823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:44.572431 24814 server_base.cc:1047] running on GCE node
W20251026 15:08:44.572391 24821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:44.572499 24820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:44.572695 24814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:44.572935 24814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:44.574126 24814 hybrid_clock.cc:648] HybridClock initialized: now 1761491324574118 us; error 62 us; skew 500 ppm
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491324, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:44.576438 24814 init.cc:377] Logged in from keytab as oryx/127.14.212.66@KRBTEST.COM (short username oryx)
I20251026 15:08:44.576947 24814 webserver.cc:492] Webserver started at http://127.14.212.66:34993/ using document root <none> and password file <none>
I20251026 15:08:44.577137 24814 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:44.577183 24814 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:44.578511 24814 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:44.579185 24830 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:44.579356 24814 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:44.579468 24814 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "8eedf209ac3e4314ab33cf804684d090"
format_stamp: "Formatted at 2025-10-26 15:08:43 on dist-test-slave-8pgz"
server_key: "0b7e521af152d34a643db9a0f99f072b"
server_key_iv: "5f8423835bea147f2d7f4607b477b248"
server_key_version: "encryptionkey@0"
I20251026 15:08:44.579761 24814 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:44.592706 24814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:44.594980 24814 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:44.595120 24814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:44.595379 24814 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:44.595711 24814 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:44.595745 24814 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:44.595783 24814 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:44.595814 24814 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:44.601755 24814 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:41273
I20251026 15:08:44.601801 24943 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:41273 every 8 connection(s)
I20251026 15:08:44.602115 24814 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:44.608050 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 24814
I20251026 15:08:44.608322 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:42007
--local_ip_for_outbound_sockets=127.14.212.67
--tserver_master_addrs=127.14.212.126:33551
--webserver_port=38243
--webserver_interface=127.14.212.67
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.212.67
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/oryx_127.14.212.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:08:44.608952 24815 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:44.602823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:57377 (local address 127.14.212.126:33551)
1026 15:08:44.602879 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:44.602880 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:44.603018 (+ 138us) server_negotiation.cc:408] Connection header received
1026 15:08:44.603489 (+ 471us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:44.603490 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:44.603500 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:44.603572 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:44.604339 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.604853 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.605839 (+ 986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.605950 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.608901 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:57377: BlockingRecv error: recv got EOF from 127.14.212.66:57377 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20251026 15:08:44.608968 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:44.602775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:57377)
1026 15:08:44.602926 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:44.602957 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:44.603413 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:44.603627 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:44.603630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:44.603932 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:44.604242 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.604245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.604937 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.604938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:44.605774 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.605776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.605883 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.606442 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:44.606460 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:44.608796 (+ 2336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
W20251026 15:08:44.609166 24944 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:33551 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.212.126:33551: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:44.613468 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:44.609384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:51333)
1026 15:08:44.609429 (+ 45us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:44.609441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:44.609483 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:44.609603 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:44.609604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:44.609735 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:44.609966 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.609969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.610660 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.610661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:44.611017 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.611018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.611083 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.611523 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:44.611527 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:44.613421 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20251026 15:08:44.613631 24815 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:44.609459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:51333 (local address 127.14.212.126:33551)
1026 15:08:44.609496 (+ 37us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:44.609498 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:44.609506 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:08:44.609531 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:44.609531 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:44.609543 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:44.609594 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:44.610037 (+ 443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.610584 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.611086 (+ 502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.611280 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.613552 (+ 2272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:51333: BlockingRecv error: recv got EOF from 127.14.212.66:51333 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:44.618026 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:44.613756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:57507)
1026 15:08:44.613798 (+ 42us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:44.613808 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:44.613841 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:44.613990 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:44.613992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:44.614138 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:44.614259 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.614260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.615266 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.615267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:44.615610 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.615611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.615671 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.616151 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:44.616155 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:44.617977 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20251026 15:08:44.618156 24815 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:44.613808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:57507 (local address 127.14.212.126:33551)
1026 15:08:44.613839 (+ 31us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:44.613841 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:44.613849 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:08:44.613880 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:44.613881 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:44.613895 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:44.613965 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:44.614344 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.615171 (+ 827us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.615698 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.615901 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.618099 (+ 2198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:57507: BlockingRecv error: recv got EOF from 127.14.212.66:57507 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20251026 15:08:44.618217 24944 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251026 15:08:44.689533 24947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:44.689708 24947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:44.689731 24947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:44.691123 24947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:44.691164 24947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:44.691190 24947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:08:44.692728 24947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/krb5kdc/oryx_127.14.212.67.keytab
--principal=oryx/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:42007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=38243
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:44.692924 24947 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:44.693069 24947 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:08:44.695709 24953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:44.695798 24955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:44.695747 24952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:44.695969 24947 server_base.cc:1047] running on GCE node
I20251026 15:08:44.696163 24947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:44.696368 24947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:44.697500 24947 hybrid_clock.cc:648] HybridClock initialized: now 1761491324697484 us; error 29 us; skew 500 ppm
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491324, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:08:44.699755 24947 init.cc:377] Logged in from keytab as oryx/127.14.212.67@KRBTEST.COM (short username oryx)
I20251026 15:08:44.700263 24947 webserver.cc:492] Webserver started at http://127.14.212.67:38243/ using document root <none> and password file <none>
I20251026 15:08:44.700441 24947 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:44.700493 24947 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:44.701797 24947 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:08:44.702368 24962 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:44.702548 24947 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:08:44.702622 24947 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "ca8990ac7bd1465080f713cd3488e060"
format_stamp: "Formatted at 2025-10-26 15:08:44 on dist-test-slave-8pgz"
server_key: "8dc7d3888c0665e3ed9e9afbfdbbaf3f"
server_key_iv: "f90876394c3d1f1124c81174fa47dd74"
server_key_version: "encryptionkey@0"
I20251026 15:08:44.702895 24947 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:44.709714 24947 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:44.711687 24947 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:44.711820 24947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:44.712028 24947 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:08:44.712318 24947 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:08:44.712352 24947 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:44.712385 24947 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:08:44.712405 24947 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:44.718065 24947 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:42007
I20251026 15:08:44.718132 25075 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:42007 every 8 connection(s)
I20251026 15:08:44.718505 24947 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:08:44.723060 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 24947
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
W20251026 15:08:44.725787 24815 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:44.719376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:48529 (local address 127.14.212.126:33551)
1026 15:08:44.719451 (+ 75us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:44.719454 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:44.719618 (+ 164us) server_negotiation.cc:408] Connection header received
1026 15:08:44.720087 (+ 469us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:44.720088 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:44.720102 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:44.720172 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:44.720953 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.721463 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.722505 (+ 1042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.722632 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.725743 (+ 3111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:48529: BlockingRecv error: recv got EOF from 127.14.212.67:48529 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20251026 15:08:44.725804 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:44.719332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:48529)
1026 15:08:44.719516 (+ 184us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:44.719543 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:44.719987 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:44.720255 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:44.720258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:44.720575 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:44.720859 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.720862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.721543 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.721544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:44.722360 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.722362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.722545 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.723091 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:44.723109 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:44.725637 (+ 2528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
W20251026 15:08:44.726009 25076 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:33551 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.212.126:33551: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:44.730582 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:44.726219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:48715)
1026 15:08:44.726419 (+ 200us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:44.726429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:44.726474 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:44.726764 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:44.726765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:44.726880 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:44.727028 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.727030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.727789 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.727790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:44.728258 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.728261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.728327 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.728718 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:44.728723 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:44.730513 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159}
W20251026 15:08:44.730661 24815 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:44.726303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:48715 (local address 127.14.212.126:33551)
1026 15:08:44.726337 (+ 34us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:44.726339 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:44.726464 (+ 125us) server_negotiation.cc:408] Connection header received
1026 15:08:44.726654 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:44.726655 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:44.726669 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:44.726732 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:44.727108 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.727714 (+ 606us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.728337 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.728533 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.730604 (+ 2071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:48715: BlockingRecv error: recv got EOF from 127.14.212.67:48715 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20251026 15:08:44.730933 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:44.723557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37248 (local address 127.14.212.126:33551)
1026 15:08:44.723610 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:44.723612 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:44.723708 (+ 96us) server_negotiation.cc:408] Connection header received
1026 15:08:44.723754 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:44.723755 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:44.723768 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:44.723825 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:44.724655 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.725282 (+ 627us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.726077 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.726278 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.727437 (+ 1159us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:44.727443 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:44.727444 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:08:44.727469 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:44.729460 (+ 1991us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:44.729780 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:44.729784 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:44.729784 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:44.729861 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:08:44.730036 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:08:44.730036 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:08:44.730037 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:08:44.730356 (+ 319us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:44.730602 (+ 246us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:44.730698 (+ 96us) server_negotiation.cc:300] Negotiation successful
1026 15:08:44.730842 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:44 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:44.734577 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:44.730826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:48297)
1026 15:08:44.730959 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:44.730969 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:44.731020 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:44.731203 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:44.731204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:44.731329 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:44.731446 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.731448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.732220 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.732221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:44.732571 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:44.732572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.732635 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.733056 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:44.733060 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:44.734531 (+ 1471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
W20251026 15:08:44.734681 24815 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:44.730884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:48297 (local address 127.14.212.126:33551)
1026 15:08:44.730920 (+ 36us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:44.730921 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:44.731006 (+ 85us) server_negotiation.cc:408] Connection header received
1026 15:08:44.731078 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:44.731079 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:44.731090 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:44.731162 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:44.731516 (+ 354us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.732132 (+ 616us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:44.732670 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:44.732791 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:44.734612 (+ 1821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:48297: BlockingRecv error: recv got EOF from 127.14.212.67:48297 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20251026 15:08:44.734706 25076 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Oct 26 15:08:45 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:45 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:45.476739 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:45.472353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:49477)
1026 15:08:45.472506 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:45.472516 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:45.472568 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:45.472699 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:45.472700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:45.472871 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:45.473059 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:45.473061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:45.473776 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:45.473777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:45.474238 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:45.474240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:45.474302 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:45.474868 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:45.474873 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:45.476673 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
W20251026 15:08:45.476872 25082 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:45.472455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:49477 (local address 127.14.212.126:33551)
1026 15:08:45.472574 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:45.472576 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:45.472584 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:08:45.472608 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:45.472609 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:45.472618 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:45.472685 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:45.473159 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:45.473692 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:45.474341 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:45.474529 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:45.476790 (+ 2261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:49477: BlockingRecv error: recv got EOF from 127.14.212.65:49477 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
Oct 26 15:08:45 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:45 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:45.622812 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:45.618674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:53353)
1026 15:08:45.618815 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:45.618826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:45.618895 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:45.619021 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:45.619021 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:45.619193 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:45.619387 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:45.619389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:45.620010 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:45.620010 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:45.620449 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:45.620451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:45.620524 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:45.621074 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:45.621078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:45.622749 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
W20251026 15:08:45.622988 25082 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:45.618755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:53353 (local address 127.14.212.126:33551)
1026 15:08:45.618818 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:45.618820 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:45.618888 (+ 68us) server_negotiation.cc:408] Connection header received
1026 15:08:45.618948 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:45.618949 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:45.618961 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:45.619009 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:45.619457 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:45.619945 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:45.620526 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:45.620658 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:45.622894 (+ 2236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:53353: BlockingRecv error: recv got EOF from 127.14.212.66:53353 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Oct 26 15:08:45 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:45 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:45.739452 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:45.735077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:54713)
1026 15:08:45.735249 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:45.735260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:45.735331 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:45.735533 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:45.735534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:45.735724 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:45.735879 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:45.735882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:45.736620 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:45.736621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:45.737027 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:45.737029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:45.737094 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:45.737639 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:45.737643 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:45.739385 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
W20251026 15:08:45.739553 25082 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:45.735138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:54713 (local address 127.14.212.126:33551)
1026 15:08:45.735225 (+ 87us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:45.735227 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:45.735318 (+ 91us) server_negotiation.cc:408] Connection header received
1026 15:08:45.735417 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:45.735418 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:45.735433 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:45.735509 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:45.735985 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:45.736527 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:45.737134 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:45.737291 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:45.739488 (+ 2197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:54713: BlockingRecv error: recv got EOF from 127.14.212.67:54713 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Oct 26 15:08:46 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:46 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:46.485800 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:46.477216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:43395)
1026 15:08:46.477345 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:46.477357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:46.477407 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:46.477562 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:46.477563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:46.477731 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:46.477892 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:46.477893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:46.478628 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:46.478629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:46.479004 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:46.479005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:46.479060 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:46.483817 (+ 4757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:46.483823 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:46.485717 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
W20251026 15:08:46.485942 25086 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:46.477330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:43395 (local address 127.14.212.126:33551)
1026 15:08:46.477450 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:46.477452 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:46.477459 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:08:46.477479 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:46.477480 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:46.477489 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:46.477550 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:46.477970 (+ 420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:46.478543 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:46.479064 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:46.479172 (+ 108us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:46.485862 (+ 6690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:43395: BlockingRecv error: recv got EOF from 127.14.212.65:43395 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
Oct 26 15:08:46 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:46 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:46.627270 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:46.623295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:37443)
1026 15:08:46.623424 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:46.623434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:46.623483 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:46.623608 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:46.623609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:46.623765 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:46.623925 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:46.623926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:46.624587 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:46.624587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:46.624936 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:46.624937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:46.624991 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:46.625519 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:46.625523 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:46.627213 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
W20251026 15:08:46.627390 25086 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:46.623395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:37443 (local address 127.14.212.126:33551)
1026 15:08:46.623459 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:46.623461 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:46.623471 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:46.623530 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:46.623531 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:46.623541 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:46.623596 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:46.623996 (+ 400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:46.624524 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:46.625023 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:46.625191 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:46.627318 (+ 2127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:37443: BlockingRecv error: recv got EOF from 127.14.212.66:37443 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Oct 26 15:08:46 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:46 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:46.744441 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:46.740007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:46609)
1026 15:08:46.740196 (+ 189us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:46.740210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:46.740279 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:46.740507 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:46.740508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:46.740684 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:46.740841 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:46.740843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:46.741644 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:46.741645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:46.742030 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:46.742032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:46.742098 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:46.742678 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:46.742682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:46.744350 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
W20251026 15:08:46.744505 25086 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:46.740189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:46609 (local address 127.14.212.126:33551)
1026 15:08:46.740250 (+ 61us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:46.740252 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:46.740261 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:08:46.740378 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:46.740379 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:46.740395 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:46.740455 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:46.740959 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:46.741564 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:46.742156 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:46.742355 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:46.744449 (+ 2094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:46609: BlockingRecv error: recv got EOF from 127.14.212.67:46609 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Oct 26 15:08:47 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:47 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:47.490629 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:47.486364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:38339)
1026 15:08:47.486536 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:47.486551 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:47.486611 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:47.486758 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:47.486759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:47.486927 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:47.487083 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:47.487085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:47.487764 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:47.487765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:47.488225 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:47.488227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:47.488304 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:47.488830 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:47.488834 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:47.490548 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
W20251026 15:08:47.490804 25090 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:47.486499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:38339 (local address 127.14.212.126:33551)
1026 15:08:47.486627 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:47.486629 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:47.486641 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:08:47.486672 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:47.486673 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:47.486685 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:47.486746 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:47.487169 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:47.487674 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:47.488320 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:47.488529 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:47.490720 (+ 2191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:38339: BlockingRecv error: recv got EOF from 127.14.212.65:38339 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
Oct 26 15:08:47 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:47 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:47.632200 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:47.627839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:39219)
1026 15:08:47.627987 (+ 148us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:47.628002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:47.628076 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:47.628217 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:47.628217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:47.628385 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:47.628560 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:47.628562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:47.629272 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:47.629273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:47.629771 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:47.629774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:47.629839 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:47.630420 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:47.630428 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:47.632135 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
W20251026 15:08:47.632300 25090 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:47.627954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:39219 (local address 127.14.212.126:33551)
1026 15:08:47.628013 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:47.628015 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:47.628065 (+ 50us) server_negotiation.cc:408] Connection header received
1026 15:08:47.628141 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:47.628142 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:47.628152 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:47.628204 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:47.628646 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:47.629192 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:47.629840 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:47.630036 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:47.632232 (+ 2196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:39219: BlockingRecv error: recv got EOF from 127.14.212.66:39219 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Oct 26 15:08:47 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:47 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:47.749419 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:47.745146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:56915)
1026 15:08:47.745322 (+ 176us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:47.745336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:47.745414 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:47.745548 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:47.745549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:47.745734 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:47.745885 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:47.745887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:47.746564 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:47.746565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:47.746931 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:47.746933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:47.746995 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:47.747617 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:47.747622 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:47.749345 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
W20251026 15:08:47.749528 25090 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:47.745257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:56915 (local address 127.14.212.126:33551)
1026 15:08:47.745327 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:47.745329 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:47.745399 (+ 70us) server_negotiation.cc:408] Connection header received
1026 15:08:47.745474 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:47.745475 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:47.745486 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:47.745536 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:47.745966 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:47.746495 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:47.746999 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:47.747163 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:47.749466 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:56915: BlockingRecv error: recv got EOF from 127.14.212.67:56915 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Oct 26 15:08:48 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:48 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:48.496016 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:48.491231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:39617)
1026 15:08:48.491357 (+ 126us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:48.491372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:48.491462 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:48.491598 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:48.491599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:48.491763 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:48.491922 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:48.491924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:48.492934 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:48.492936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:48.493362 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:48.493364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:48.493432 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:48.493960 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:48.493964 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:48.495934 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
W20251026 15:08:48.496135 25094 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:48.491343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:39617 (local address 127.14.212.126:33551)
1026 15:08:48.491474 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:48.491476 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:48.491486 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:48.491512 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:48.491513 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:48.491524 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:48.491582 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:48.492038 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:48.492858 (+ 820us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:48.493445 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:48.493635 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:48.496057 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:39617: BlockingRecv error: recv got EOF from 127.14.212.65:39617 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
Oct 26 15:08:48 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:48 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:48.637013 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:48.632767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:50351)
1026 15:08:48.632932 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:48.632945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:48.633016 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:48.633176 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:48.633177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:48.633352 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:48.633508 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:48.633510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:48.634166 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:48.634167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:48.634577 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:48.634579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:48.634645 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:48.635228 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:48.635232 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:48.636937 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":82,"threads_started":1}
W20251026 15:08:48.637101 25094 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:48.632877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:50351 (local address 127.14.212.126:33551)
1026 15:08:48.632935 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:48.632938 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:48.633015 (+ 77us) server_negotiation.cc:408] Connection header received
1026 15:08:48.633081 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:48.633082 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:48.633095 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:48.633148 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:48.633598 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:48.634090 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:48.634647 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:48.634842 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:48.637043 (+ 2201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:50351: BlockingRecv error: recv got EOF from 127.14.212.66:50351 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Oct 26 15:08:48 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:48 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:48.753831 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:48.749822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:57823)
1026 15:08:48.749941 (+ 119us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:48.749952 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:48.750012 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:48.750157 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:48.750157 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:48.750330 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:48.750522 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:48.750524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:48.751200 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:48.751201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:48.751567 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:48.751568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:48.751628 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:48.752135 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:48.752139 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:48.753777 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
W20251026 15:08:48.753921 25094 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:48.749880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:57823 (local address 127.14.212.126:33551)
1026 15:08:48.749944 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:48.749947 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:48.750003 (+ 56us) server_negotiation.cc:408] Connection header received
1026 15:08:48.750074 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:48.750075 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:48.750085 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:48.750132 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:48.750576 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:48.751101 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:48.751650 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:48.751828 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:48.753861 (+ 2033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:57823: BlockingRecv error: recv got EOF from 127.14.212.67:57823 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Oct 26 15:08:49 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:49 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:49.500402 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:49.496532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:41681)
1026 15:08:49.496659 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:49.496669 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:49.496721 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:49.496839 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:49.496840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:49.497000 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:49.497156 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:49.497158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:49.497795 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:49.497795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:49.498191 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:49.498193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:49.498247 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:49.498741 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:49.498745 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:49.500346 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
W20251026 15:08:49.500552 25098 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:49.496574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:41681 (local address 127.14.212.126:33551)
1026 15:08:49.496684 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:49.496686 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:49.496708 (+ 22us) server_negotiation.cc:408] Connection header received
1026 15:08:49.496763 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:49.496764 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:49.496774 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:49.496837 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:49.497221 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:49.497737 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:49.498255 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:49.498368 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:49.500484 (+ 2116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:41681: BlockingRecv error: recv got EOF from 127.14.212.65:41681 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
Oct 26 15:08:49 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:49 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:49.641844 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:49.637552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:50819)
1026 15:08:49.637687 (+ 135us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:49.637698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:49.637771 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:49.637928 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:49.637928 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:49.638125 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:49.638312 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:49.638315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:49.639013 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:49.639015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:49.639404 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:49.639405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:49.639469 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:49.640000 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:49.640005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:49.641769 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
W20251026 15:08:49.641942 25098 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:49.637688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:50819 (local address 127.14.212.126:33551)
1026 15:08:49.637760 (+ 72us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:49.637763 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:49.637773 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:49.637851 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:49.637852 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:49.637862 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:49.637904 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:49.638444 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:49.638938 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:49.639477 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:49.639672 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:49.641880 (+ 2208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:50819: BlockingRecv error: recv got EOF from 127.14.212.66:50819 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Oct 26 15:08:49 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:49 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:49.758692 25100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:49.754457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:48021)
1026 15:08:49.754599 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:49.754609 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:49.754678 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:49.754844 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:49.754845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:49.755011 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:49.755202 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:49.755204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:49.755879 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:49.755880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:49.756258 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:49.756259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:49.756312 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:49.756874 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:49.756879 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:49.758635 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
W20251026 15:08:49.758821 25098 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:49.754530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:48021 (local address 127.14.212.126:33551)
1026 15:08:49.754585 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:49.754586 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:49.754662 (+ 76us) server_negotiation.cc:408] Connection header received
1026 15:08:49.754731 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:49.754732 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:49.754742 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:49.754801 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:49.755304 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:49.755809 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:49.756331 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:49.756469 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:49.758741 (+ 2272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:48021: BlockingRecv error: recv got EOF from 127.14.212.67:48021 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Oct 26 15:08:50 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:50 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:50.505105 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:50.500959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:47249)
1026 15:08:50.501084 (+ 125us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:50.501095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:50.501176 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:50.501366 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:50.501368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:50.501534 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:50.501701 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:50.501703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:50.502376 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:50.502377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:50.502818 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:50.502820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:50.502888 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:50.503393 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:50.503398 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:50.505049 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
W20251026 15:08:50.505221 25102 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:50.501061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:47249 (local address 127.14.212.126:33551)
1026 15:08:50.501204 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:50.501205 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:50.501216 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:08:50.501241 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:50.501241 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:50.501252 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:50.501324 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:50.501774 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:50.502294 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:50.502883 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:50.502997 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:50.505150 (+ 2153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:47249: BlockingRecv error: recv got EOF from 127.14.212.65:47249 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":57,"threads_started":1}
Oct 26 15:08:50 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:50 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:50.646538 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:50.642450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:41315)
1026 15:08:50.642574 (+ 124us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:50.642584 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:50.642644 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:50.642771 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:50.642772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:50.642934 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:50.643092 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:50.643094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:50.643756 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:50.643757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:50.644121 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:50.644122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:50.644198 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:50.644704 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:50.644708 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:50.646480 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
W20251026 15:08:50.646682 25102 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:50.642492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:41315 (local address 127.14.212.126:33551)
1026 15:08:50.642540 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:50.642542 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:50.642638 (+ 96us) server_negotiation.cc:408] Connection header received
1026 15:08:50.642691 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:50.642692 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:50.642703 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:50.642759 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:50.643183 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:50.643692 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:50.644197 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:50.644305 (+ 108us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:50.646603 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:41315: BlockingRecv error: recv got EOF from 127.14.212.66:41315 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Oct 26 15:08:50 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:50 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:50.763315 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:50.759277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:51795)
1026 15:08:50.759464 (+ 187us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:50.759478 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:50.759552 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:50.759700 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:50.759702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:50.759877 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:50.760031 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:50.760032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:50.760679 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:50.760681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:50.761063 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:50.761065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:50.761137 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:50.761652 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:50.761656 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:50.763238 (+ 1582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
W20251026 15:08:50.763420 25102 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:50.759384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:51795 (local address 127.14.212.126:33551)
1026 15:08:50.759445 (+ 61us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:50.759448 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:50.759544 (+ 96us) server_negotiation.cc:408] Connection header received
1026 15:08:50.759620 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:50.759621 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:50.759632 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:50.759681 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:50.760110 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:50.760604 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:50.761136 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:50.761338 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:50.763350 (+ 2012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:51795: BlockingRecv error: recv got EOF from 127.14.212.67:51795 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Oct 26 15:08:51 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:51 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:51.513249 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:51.505699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:33451)
1026 15:08:51.505870 (+ 171us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:51.505883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:51.505957 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:51.506107 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:51.506108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:51.506298 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:51.506497 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:51.506500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:51.507386 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:51.507387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:51.507759 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:51.507761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:51.507820 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:51.511536 (+ 3716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:51.511541 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:51.513162 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
W20251026 15:08:51.513355 25106 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:51.505831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:33451 (local address 127.14.212.126:33551)
1026 15:08:51.505951 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:51.505953 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:51.505963 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:51.506017 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:51.506018 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:51.506028 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:51.506090 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:51.506584 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:51.507312 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:51.507838 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:51.507983 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:51.513277 (+ 5294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:33451: BlockingRecv error: recv got EOF from 127.14.212.65:33451 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
Oct 26 15:08:51 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:51 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:51.651392 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:51.647050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:59333)
1026 15:08:51.647240 (+ 190us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:51.647256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:51.647347 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:51.647573 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:51.647574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:51.647757 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:51.647917 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:51.647919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:51.648596 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:51.648597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:51.648958 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:51.648959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:51.649013 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:51.649537 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:51.649542 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:51.651307 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":123,"threads_started":1}
W20251026 15:08:51.651459 25106 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:51.647209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:59333 (local address 127.14.212.126:33551)
1026 15:08:51.647254 (+ 45us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:51.647256 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:51.647380 (+ 124us) server_negotiation.cc:408] Connection header received
1026 15:08:51.647429 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:51.647430 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:51.647444 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:51.647496 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:51.647996 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:51.648530 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:51.649019 (+ 489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:51.649138 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:51.651406 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:59333: BlockingRecv error: recv got EOF from 127.14.212.66:59333 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Oct 26 15:08:51 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:51 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:51.768358 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:51.763923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:41857)
1026 15:08:51.764069 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:51.764082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:51.764150 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:51.764286 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:51.764287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:51.764483 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:51.764637 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:51.764639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:51.765293 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:51.765294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:51.765682 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:51.765684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:51.765750 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:51.766319 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:51.766323 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:51.768303 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
W20251026 15:08:51.768478 25106 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:51.764038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:41857 (local address 127.14.212.126:33551)
1026 15:08:51.764155 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:51.764158 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:51.764186 (+ 28us) server_negotiation.cc:408] Connection header received
1026 15:08:51.764214 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:51.764215 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:51.764226 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:51.764273 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:51.764723 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:51.765216 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:51.765773 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:51.765959 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:51.768396 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:41857: BlockingRecv error: recv got EOF from 127.14.212.67:41857 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Oct 26 15:08:52 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:52 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:52.518718 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:52.513826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:35933)
1026 15:08:52.513983 (+ 157us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:52.513996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:52.514054 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:52.514239 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:52.514241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:52.514623 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:52.514843 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:52.514847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:52.515597 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:52.515598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:52.516176 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:52.516179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:52.516258 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:52.516784 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:52.516788 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:52.518660 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
W20251026 15:08:52.518855 25110 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:52.513933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:35933 (local address 127.14.212.126:33551)
1026 15:08:52.514057 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:52.514059 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:52.514070 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:08:52.514133 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:52.514134 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:52.514147 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:52.514222 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:52.514981 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:52.515505 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:52.516293 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:52.516468 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:52.518775 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:35933: BlockingRecv error: recv got EOF from 127.14.212.65:35933 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
Oct 26 15:08:52 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:52 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:52.656472 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:52.651943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:42661)
1026 15:08:52.652106 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:52.652156 (+ 50us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:52.652232 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:52.652378 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:52.652379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:52.652549 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:52.652704 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:52.652706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:52.653551 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:52.653552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:52.653932 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:52.653934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:52.653995 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:52.654676 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:52.654681 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:52.656384 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
W20251026 15:08:52.656550 25110 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:52.652034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:42661 (local address 127.14.212.126:33551)
1026 15:08:52.652120 (+ 86us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:52.652122 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:52.652240 (+ 118us) server_negotiation.cc:408] Connection header received
1026 15:08:52.652274 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:52.652275 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:52.652286 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:52.652342 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:52.652783 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:52.653472 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:52.654103 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:52.654298 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:52.656488 (+ 2190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:42661: BlockingRecv error: recv got EOF from 127.14.212.66:42661 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Oct 26 15:08:52 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:52 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:52.772866 25112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:52.768811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:53577)
1026 15:08:52.768947 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:52.768958 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:52.769018 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:52.769191 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:52.769192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:52.769356 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:52.769517 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:52.769519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:52.770196 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:52.770198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:52.770614 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:52.770616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:52.770679 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:52.771230 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:52.771234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:52.772810 (+ 1576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
W20251026 15:08:52.773054 25110 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:52.768866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:53577 (local address 127.14.212.126:33551)
1026 15:08:52.768916 (+ 50us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:52.768917 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:52.769016 (+ 99us) server_negotiation.cc:408] Connection header received
1026 15:08:52.769061 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:52.769061 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:52.769070 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:52.769129 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:52.769598 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:52.770075 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:52.770677 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:52.770782 (+ 105us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:52.772946 (+ 2164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:53577: BlockingRecv error: recv got EOF from 127.14.212.67:53577 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Oct 26 15:08:53 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:53 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:53.523345 25113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:53.519218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:35967)
1026 15:08:53.519379 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:53.519393 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:53.519475 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:53.519603 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:53.519604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:53.519780 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:53.519943 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:53.519945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:53.520622 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:53.520623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:53.521002 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:53.521003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:53.521065 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:53.521574 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:53.521578 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:53.523286 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
W20251026 15:08:53.523465 25114 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:53.519337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:35967 (local address 127.14.212.126:33551)
1026 15:08:53.519463 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:53.519466 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:53.519476 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:53.519523 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:53.519524 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:53.519533 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:53.519594 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:53.519998 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:53.520529 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:53.521071 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:53.521177 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:53.523394 (+ 2217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:35967: BlockingRecv error: recv got EOF from 127.14.212.65:35967 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
Oct 26 15:08:53 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:53 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:53.661880 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:53.657013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:55837)
1026 15:08:53.657148 (+ 135us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:53.657158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:53.657223 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:53.657348 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:53.657349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:53.657528 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:53.657682 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:53.657684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:53.658603 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:53.658605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:53.659147 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:53.659149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:53.659230 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:53.659800 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:53.659806 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:53.661798 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
W20251026 15:08:53.661954 25114 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:53.657055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:55837 (local address 127.14.212.126:33551)
1026 15:08:53.657102 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:53.657105 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:53.657216 (+ 111us) server_negotiation.cc:408] Connection header received
1026 15:08:53.657278 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:53.657279 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:53.657291 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:53.657339 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:53.657760 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:53.658526 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:53.659236 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:53.659470 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:53.661895 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:55837: BlockingRecv error: recv got EOF from 127.14.212.66:55837 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Oct 26 15:08:53 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:53 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:53.777777 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:53.773500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:36769)
1026 15:08:53.773646 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:53.773656 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:53.773712 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:53.773926 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:53.773927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:53.774092 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:53.774284 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:53.774286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:53.774936 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:53.774937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:53.775344 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:53.775346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:53.775411 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:53.775964 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:53.775969 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:53.777722 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
W20251026 15:08:53.777890 25114 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:53.773619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:36769 (local address 127.14.212.126:33551)
1026 15:08:53.773707 (+ 88us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:53.773709 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:53.773721 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:08:53.773829 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:53.773831 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:53.773843 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:53.773911 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:53.774362 (+ 451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:53.774861 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:53.775420 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:53.775537 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:53.777814 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:36769: BlockingRecv error: recv got EOF from 127.14.212.67:36769 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Oct 26 15:08:54 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:54 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:54.528285 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:54.523876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:60701)
1026 15:08:54.524043 (+ 167us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:54.524054 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:54.524121 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:54.524300 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:54.524301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:54.524540 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:54.524769 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:54.524772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:54.525448 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:54.525450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:54.525832 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:54.525834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:54.525899 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:54.526465 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:54.526470 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:54.528227 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":65,"threads_started":1}
W20251026 15:08:54.528401 25118 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:54.523987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:60701 (local address 127.14.212.126:33551)
1026 15:08:54.524111 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:54.524113 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:54.524125 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:08:54.524194 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:54.524195 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:54.524208 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:54.524280 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:54.524836 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:54.525349 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:54.525926 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:54.526065 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:54.528332 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:60701: BlockingRecv error: recv got EOF from 127.14.212.65:60701 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":50,"threads_started":1}
Oct 26 15:08:54 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:54 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:54.666868 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:54.662452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:39285)
1026 15:08:54.662580 (+ 128us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:54.662590 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:54.662660 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:54.662800 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:54.662801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:54.662968 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:54.663131 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:54.663133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:54.663899 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:54.663900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:54.664322 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:54.664324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:54.664386 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:54.664892 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:54.664896 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:54.666786 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
W20251026 15:08:54.666949 25118 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:54.662515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:39285 (local address 127.14.212.126:33551)
1026 15:08:54.662571 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:54.662573 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:54.662682 (+ 109us) server_negotiation.cc:408] Connection header received
1026 15:08:54.662716 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:54.662717 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:54.662729 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:54.662789 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:54.663239 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:54.663801 (+ 562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:54.664393 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:54.664522 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:54.666881 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:39285: BlockingRecv error: recv got EOF from 127.14.212.66:39285 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Oct 26 15:08:54 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:54 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:54.782685 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:54.778293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:46059)
1026 15:08:54.778459 (+ 166us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:54.778469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:54.778534 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:54.778663 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:54.778664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:54.778837 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:54.778994 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:54.778996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:54.779630 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:54.779631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:54.779998 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:54.779999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:54.780062 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:54.780680 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:54.780685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:54.782630 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":66,"threads_started":1}
W20251026 15:08:54.782814 25118 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:54.778458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:46059 (local address 127.14.212.126:33551)
1026 15:08:54.778526 (+ 68us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:54.778529 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:54.778539 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:54.778594 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:54.778595 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:54.778606 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:54.778649 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:54.779075 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:54.779555 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:54.780071 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:54.780269 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:54.782751 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:46059: BlockingRecv error: recv got EOF from 127.14.212.67:46059 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Oct 26 15:08:55 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:55 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:55.533212 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:55.528830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:56891)
1026 15:08:55.528994 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:55.529013 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:55.529078 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:55.529216 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:55.529216 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:55.529391 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:55.529555 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:55.529557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:55.530253 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:55.530254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:55.530665 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:55.530667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:55.530723 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:55.531324 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:55.531328 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:55.533137 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
W20251026 15:08:55.533340 25122 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:55.528923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:56891 (local address 127.14.212.126:33551)
1026 15:08:55.529082 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:55.529084 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:55.529095 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:08:55.529122 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:55.529122 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:55.529133 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:55.529204 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:55.529646 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:55.530177 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:55.530742 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:55.530873 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:55.533270 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:56891: BlockingRecv error: recv got EOF from 127.14.212.65:56891 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
Oct 26 15:08:55 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:55 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:55.671599 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:55.667400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:54225)
1026 15:08:55.667547 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:55.667557 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:55.667622 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:55.667862 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:55.667863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:55.668029 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:55.668184 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:55.668185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:55.668839 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:55.668839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:55.669194 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:55.669196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:55.669246 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:55.669778 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:55.669782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:55.671542 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
W20251026 15:08:55.671729 25122 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:55.667511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:54225 (local address 127.14.212.126:33551)
1026 15:08:55.667575 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:55.667577 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:55.667612 (+ 35us) server_negotiation.cc:408] Connection header received
1026 15:08:55.667767 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:55.667769 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:55.667782 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:55.667851 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:55.668268 (+ 417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:55.668774 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:55.669269 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:55.669426 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:55.671651 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:54225: BlockingRecv error: recv got EOF from 127.14.212.66:54225 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Oct 26 15:08:55 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:55 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:55.787448 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:55.783279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:43499)
1026 15:08:55.783424 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:55.783438 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:55.783514 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:55.783647 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:55.783648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:55.783872 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:55.784029 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:55.784031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:55.784691 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:55.784692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:55.785054 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:55.785055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:55.785114 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:55.785653 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:55.785657 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:55.787372 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
W20251026 15:08:55.787559 25122 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:55.783330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:43499 (local address 127.14.212.126:33551)
1026 15:08:55.783378 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:55.783380 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:55.783503 (+ 123us) server_negotiation.cc:408] Connection header received
1026 15:08:55.783572 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:55.783573 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:55.783585 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:55.783642 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:55.784108 (+ 466us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:55.784622 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:55.785120 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:55.785241 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:55.787493 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:43499: BlockingRecv error: recv got EOF from 127.14.212.67:43499 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Oct 26 15:08:56 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:56 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:56.542153 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:56.533792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:49315)
1026 15:08:56.533939 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:56.533951 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:56.534010 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:56.534200 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:56.534202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:56.534427 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:56.534601 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:56.534603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:56.535484 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:56.535485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:56.535883 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:56.535885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:56.535948 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:56.540149 (+ 4201us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:56.540156 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:56.542074 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
W20251026 15:08:56.542321 25126 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:56.533898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:49315 (local address 127.14.212.126:33551)
1026 15:08:56.534044 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:56.534045 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:56.534055 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:08:56.534082 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:56.534083 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:56.534093 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:56.534167 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:56.534682 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:56.535388 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:56.536041 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:56.536281 (+ 240us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:56.542258 (+ 5977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:49315: BlockingRecv error: recv got EOF from 127.14.212.65:49315 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
Oct 26 15:08:56 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:56 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:56.676334 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:56.672131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:57795)
1026 15:08:56.672270 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:56.672280 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:56.672331 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:56.672489 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:56.672490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:56.672658 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:56.672814 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:56.672816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:56.673541 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:56.673542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:56.673910 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:56.673911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:56.673963 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:56.674577 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:56.674581 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:56.676278 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
W20251026 15:08:56.676502 25126 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:56.672226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:57795 (local address 127.14.212.126:33551)
1026 15:08:56.672317 (+ 91us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:56.672319 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:56.672331 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:08:56.672400 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:56.672401 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:56.672412 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:56.672469 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:56.672926 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:56.673470 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:56.673996 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:56.674153 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:56.676430 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:57795: BlockingRecv error: recv got EOF from 127.14.212.66:57795 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Oct 26 15:08:56 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:56 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:56.792476 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:56.788006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:47587)
1026 15:08:56.788174 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:56.788187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:56.788259 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:56.788494 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:56.788496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:56.788683 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:56.788888 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:56.788894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:56.789612 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:56.789613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:56.789989 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:56.789990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:56.790048 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:56.790662 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:56.790667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:56.792417 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
W20251026 15:08:56.792620 25126 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:56.788078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:47587 (local address 127.14.212.126:33551)
1026 15:08:56.788157 (+ 79us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:56.788159 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:56.788332 (+ 173us) server_negotiation.cc:408] Connection header received
1026 15:08:56.788362 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:56.788363 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:56.788375 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:56.788449 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:56.788998 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:56.789535 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:56.790089 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:56.790341 (+ 252us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:56.792549 (+ 2208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:47587: BlockingRecv error: recv got EOF from 127.14.212.67:47587 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Oct 26 15:08:57 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:57 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:57.547338 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:57.542773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:47389)
1026 15:08:57.542999 (+ 226us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:57.543015 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:57.543092 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:57.543245 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:57.543246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:57.543424 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:57.543581 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:57.543583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:57.544419 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:57.544421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:57.544938 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:57.544941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:57.544998 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:57.545526 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:57.545560 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:57.547280 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
W20251026 15:08:57.547498 25130 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:57.542882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:47389 (local address 127.14.212.126:33551)
1026 15:08:57.543017 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:57.543019 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:57.543086 (+ 67us) server_negotiation.cc:408] Connection header received
1026 15:08:57.543150 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:57.543151 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:57.543162 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:57.543232 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:57.543680 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:57.544308 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:57.545011 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:57.545132 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:57.547403 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:47389: BlockingRecv error: recv got EOF from 127.14.212.65:47389 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
Oct 26 15:08:57 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:57 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:57.681154 25131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:57.676929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:60047)
1026 15:08:57.677059 (+ 130us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:57.677073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:57.677138 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:57.677269 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:57.677270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:57.677461 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:57.677614 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:57.677616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:57.678286 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:57.678287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:57.678699 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:57.678701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:57.678768 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:57.679356 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:57.679360 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:57.681079 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
W20251026 15:08:57.681248 25130 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:57.677019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:60047 (local address 127.14.212.126:33551)
1026 15:08:57.677064 (+ 45us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:57.677066 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:57.677117 (+ 51us) server_negotiation.cc:408] Connection header received
1026 15:08:57.677192 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:57.677193 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:57.677205 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:57.677257 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:57.677705 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:57.678203 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:57.678788 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:57.678991 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:57.681192 (+ 2201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:60047: BlockingRecv error: recv got EOF from 127.14.212.66:60047 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Oct 26 15:08:57 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:57 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:57.797307 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:57.793048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:56817)
1026 15:08:57.793239 (+ 191us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:57.793255 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:57.793321 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:57.793480 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:57.793481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:57.793646 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:57.793805 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:57.793807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:57.794502 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:57.794503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:57.794866 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:57.794868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:57.794927 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:57.795481 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:57.795485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:57.797203 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
W20251026 15:08:57.797528 25130 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:57.793178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:56817 (local address 127.14.212.126:33551)
1026 15:08:57.793254 (+ 76us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:57.793256 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:57.793318 (+ 62us) server_negotiation.cc:408] Connection header received
1026 15:08:57.793392 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:57.793392 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:57.793404 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:57.793469 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:57.793880 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:57.794429 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:57.794938 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:57.795053 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:57.797342 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:56817: BlockingRecv error: recv got EOF from 127.14.212.67:56817 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Oct 26 15:08:58 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:58 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:58.552690 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:58.547881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:37849)
1026 15:08:58.548017 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:58.548031 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:58.548104 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:58.548256 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:58.548256 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:58.548425 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:58.548595 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:58.548597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:58.549231 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:58.549232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:58.549615 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:58.549616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:58.549679 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:58.550455 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:58.550461 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:58.552631 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
W20251026 15:08:58.552806 25134 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:58.547957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:37849 (local address 127.14.212.126:33551)
1026 15:08:58.548094 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:58.548096 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:58.548107 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:08:58.548155 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:58.548156 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:58.548170 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:58.548241 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:58.548657 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:58.549162 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:58.549691 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:58.549818 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:58.552741 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:37849: BlockingRecv error: recv got EOF from 127.14.212.65:37849 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
Oct 26 15:08:58 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:58 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:58.686883 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:58.681842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:39089)
1026 15:08:58.682002 (+ 160us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:58.682017 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:58.682098 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:58.682310 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:58.682311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:58.682559 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:58.682722 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:58.682724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:58.683884 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:58.683885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:58.684274 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:58.684275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:58.684337 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:58.684862 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:58.684866 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:58.686803 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
W20251026 15:08:58.686975 25134 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:58.681952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:39089 (local address 127.14.212.126:33551)
1026 15:08:58.682028 (+ 76us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:58.682031 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:58.682085 (+ 54us) server_negotiation.cc:408] Connection header received
1026 15:08:58.682219 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:58.682220 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:58.682233 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:58.682283 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:58.682843 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:58.683798 (+ 955us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:58.684365 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:58.684496 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:58.686912 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:39089: BlockingRecv error: recv got EOF from 127.14.212.66:39089 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Oct 26 15:08:58 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:58 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.67@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:58.802174 25136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:58.797959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.67:47891)
1026 15:08:58.798103 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:58.798117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:58.798187 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:58.798366 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:58.798367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:58.798565 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:58.798722 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:58.798724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:58.799383 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:58.799384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:58.799767 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:58.799768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:58.799825 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:58.800388 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:58.800393 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:58.802120 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
W20251026 15:08:58.802424 25134 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:58.798108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:47891 (local address 127.14.212.126:33551)
1026 15:08:58.798234 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:58.798237 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:58.798248 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:08:58.798273 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:58.798274 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:58.798284 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:58.798341 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:58.798787 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:58.799305 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:58.799846 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:58.799964 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:58.802310 (+ 2346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.67:47891: BlockingRecv error: recv got EOF from 127.14.212.67:47891 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Oct 26 15:08:59 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:59 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.65@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:59.557425 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:59.553180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.65:50917)
1026 15:08:59.553331 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:59.553341 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:59.553390 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:59.553526 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:59.553526 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:59.553694 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:59.553856 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:59.553858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:59.554598 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:59.554598 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:59.554960 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:59.554962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:59.555019 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:59.555562 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:59.555566 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:59.557369 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
W20251026 15:08:59.557575 25138 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:59.553273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:50917 (local address 127.14.212.126:33551)
1026 15:08:59.553392 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:59.553394 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:59.553435 (+ 41us) server_negotiation.cc:408] Connection header received
1026 15:08:59.553455 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:59.553456 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:59.553465 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:59.553514 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:59.553957 (+ 443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:59.554530 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:59.555050 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:59.555190 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:59.557500 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.65:50917: BlockingRecv error: recv got EOF from 127.14.212.65:50917 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
Oct 26 15:08:59 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
Oct 26 15:08:59 dist-test-slave-8pgz krb5kdc[24069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.212.66@KRBTEST.COM for oryx/127.14.212.126@KRBTEST.COM, Server not found in Kerberos database
I20251026 15:08:59.691635 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:59.687525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33551 (local address 127.14.212.66:47559)
1026 15:08:59.687692 (+ 167us) negotiation.cc:107] Waiting for socket to connect
1026 15:08:59.687705 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:08:59.687768 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:08:59.687890 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:08:59.687891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:08:59.688053 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:08:59.688213 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:59.688215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:59.688826 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:08:59.688827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:08:59.689192 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:08:59.689194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:08:59.689248 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:59.689784 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:08:59.689787 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:08:59.691571 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.212.126:33551: Server oryx/127.14.212.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
W20251026 15:08:59.691746 25138 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:08:59.687648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:47559 (local address 127.14.212.126:33551)
1026 15:08:59.687710 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:59.687712 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:59.687762 (+ 50us) server_negotiation.cc:408] Connection header received
1026 15:08:59.687814 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:59.687815 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:59.687825 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:59.687874 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:59.688277 (+ 403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:59.688767 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:59.689259 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:59.689378 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:59.691678 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.212.66:47559: BlockingRecv error: recv got EOF from 127.14.212.66:47559 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20251026 15:08:59.726903 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 24662
I20251026 15:08:59.731441 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 24814
I20251026 15:08:59.736050 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 24947
I20251026 15:08:59.740609 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 24592
2025-10-26T15:08:59Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18444 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-10-26T15:08:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:08:59Z Disabled control of system clock
I20251026 15:08:59.768124 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:41747
--webserver_interface=127.14.212.126
--webserver_port=0
--builtin_ntp_servers=127.14.212.84:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:41747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251026 15:08:59.850098 25143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:08:59.850286 25143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:08:59.850307 25143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:08:59.851823 25143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:08:59.851868 25143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:08:59.851881 25143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:08:59.851893 25143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:08:59.851905 25143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:08:59.853492 25143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:41747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:41747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:08:59.853729 25143 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:08:59.853951 25143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:08:59.856966 25143 server_base.cc:1047] running on GCE node
W20251026 15:08:59.857007 25151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:59.857020 25149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:08:59.857045 25148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:08:59.857497 25143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:08:59.857753 25143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:08:59.858923 25143 hybrid_clock.cc:648] HybridClock initialized: now 1761491339858906 us; error 37 us; skew 500 ppm
I20251026 15:08:59.860190 25143 webserver.cc:492] Webserver started at http://127.14.212.126:35113/ using document root <none> and password file <none>
I20251026 15:08:59.860400 25143 fs_manager.cc:362] Metadata directory not provided
I20251026 15:08:59.860452 25143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:08:59.860562 25143 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:08:59.861534 25143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "88ed231201bf4507851791546770fd65"
format_stamp: "Formatted at 2025-10-26 15:08:59 on dist-test-slave-8pgz"
server_key: "6ddbf554a827256eb9ffaa4be1de4486"
server_key_iv: "a2c3923e865eed503bccf2ebb9908667"
server_key_version: "encryptionkey@0"
I20251026 15:08:59.861855 25143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "88ed231201bf4507851791546770fd65"
format_stamp: "Formatted at 2025-10-26 15:08:59 on dist-test-slave-8pgz"
server_key: "6ddbf554a827256eb9ffaa4be1de4486"
server_key_iv: "a2c3923e865eed503bccf2ebb9908667"
server_key_version: "encryptionkey@0"
I20251026 15:08:59.863176 25143 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:08:59.864037 25157 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:08:59.864279 25143 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:08:59.864360 25143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "88ed231201bf4507851791546770fd65"
format_stamp: "Formatted at 2025-10-26 15:08:59 on dist-test-slave-8pgz"
server_key: "6ddbf554a827256eb9ffaa4be1de4486"
server_key_iv: "a2c3923e865eed503bccf2ebb9908667"
server_key_version: "encryptionkey@0"
I20251026 15:08:59.864434 25143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:08:59.872962 25143 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:08:59.873219 25143 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:08:59.873318 25143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:08:59.877529 25143 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:41747
I20251026 15:08:59.877614 25209 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:41747 every 8 connection(s)
I20251026 15:08:59.877902 25143 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:08:59.878599 25210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:08:59.881057 25210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65: Bootstrap starting.
I20251026 15:08:59.881654 25210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65: Neither blocks nor log segments found. Creating new log.
I20251026 15:08:59.881912 25210 log.cc:826] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65: Log is configured to *not* fsync() on all Append() calls
I20251026 15:08:59.882354 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 25143
I20251026 15:08:59.882486 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:08:59.882710 15185 external_mini_cluster.cc:1468] Setting key 47f1df7e820d0f4493d58061cbf46eac
I20251026 15:08:59.882853 25210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65: No bootstrap required, opened a new log
I20251026 15:08:59.884773 25210 raft_consensus.cc:359] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ed231201bf4507851791546770fd65" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41747 } }
I20251026 15:08:59.884913 25210 raft_consensus.cc:385] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:08:59.884934 25210 raft_consensus.cc:740] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ed231201bf4507851791546770fd65, State: Initialized, Role: FOLLOWER
I20251026 15:08:59.885044 25210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ed231201bf4507851791546770fd65" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41747 } }
I20251026 15:08:59.885109 25210 raft_consensus.cc:399] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:08:59.885147 25210 raft_consensus.cc:493] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:08:59.885195 25210 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:08:59.885878 25210 raft_consensus.cc:515] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ed231201bf4507851791546770fd65" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41747 } }
I20251026 15:08:59.886006 25210 leader_election.cc:304] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88ed231201bf4507851791546770fd65; no voters:
I20251026 15:08:59.886165 25210 leader_election.cc:290] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:08:59.886230 25215 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:08:59.886441 25210 sys_catalog.cc:565] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:08:59.886533 25215 raft_consensus.cc:697] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [term 1 LEADER]: Becoming Leader. State: Replica: 88ed231201bf4507851791546770fd65, State: Running, Role: LEADER
I20251026 15:08:59.886660 25215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ed231201bf4507851791546770fd65" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41747 } }
I20251026 15:08:59.887005 25215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88ed231201bf4507851791546770fd65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ed231201bf4507851791546770fd65" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41747 } } }
I20251026 15:08:59.887151 25215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:59.887342 25216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88ed231201bf4507851791546770fd65. Latest consensus state: current_term: 1 leader_uuid: "88ed231201bf4507851791546770fd65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ed231201bf4507851791546770fd65" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41747 } } }
I20251026 15:08:59.887425 25224 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:08:59.887521 25216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65 [sys.catalog]: This master's current role is: LEADER
I20251026 15:08:59.888025 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:08:59.883355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.14.212.126:41747)
1026 15:08:59.883561 (+ 206us) server_negotiation.cc:207] Beginning negotiation
1026 15:08:59.883568 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:08:59.883584 (+ 16us) server_negotiation.cc:408] Connection header received
1026 15:08:59.883957 (+ 373us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:08:59.883962 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:08:59.884021 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:08:59.884136 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:08:59.884662 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:59.885573 (+ 911us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:08:59.886445 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:08:59.886709 (+ 264us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:08:59.887255 (+ 546us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:08:59.887263 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:08:59.887264 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:08:59.887504 (+ 240us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:08:59.887540 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:08:59.887549 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:08:59.887631 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1026 15:08:59.887769 (+ 138us) server_negotiation.cc:300] Negotiation successful
1026 15:08:59.887847 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20251026 15:08:59.888090 25224 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:08:59.889518 25224 catalog_manager.cc:1357] Generated new cluster ID: 9774399547cb4280b7466d0b2bba9b42
I20251026 15:08:59.889576 25224 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:08:59.899057 25224 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:08:59.899638 25224 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:08:59.908872 25224 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 88ed231201bf4507851791546770fd65: Generated new TSK 0
I20251026 15:08:59.909103 25224 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:08:59.914127 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41747
--builtin_ntp_servers=127.14.212.84:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20251026 15:09:00.002784 25234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:09:00.002995 25234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:09:00.003031 25234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:09:00.004907 25234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:09:00.004983 25234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:09:00.005028 25234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:09:00.006877 25234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:09:00.007151 25234 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:09:00.007400 25234 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:09:00.011448 25239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:00.011494 25234 server_base.cc:1047] running on GCE node
W20251026 15:09:00.011348 25242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:00.011348 25240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:00.012053 25234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:09:00.012315 25234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:09:00.013533 25234 hybrid_clock.cc:648] HybridClock initialized: now 1761491340013511 us; error 42 us; skew 500 ppm
I20251026 15:09:00.015127 25234 webserver.cc:492] Webserver started at http://127.14.212.65:46795/ using document root <none> and password file <none>
I20251026 15:09:00.015362 25234 fs_manager.cc:362] Metadata directory not provided
I20251026 15:09:00.015424 25234 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:09:00.015554 25234 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:09:00.016749 25234 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "c6b9cbe7db08476e8b2e9688caddda61"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "8b23aee26e0c67a77df97e659951f469"
server_key_iv: "98076ca9e4060071f1fe49d8afb8e5b4"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.017087 25234 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "c6b9cbe7db08476e8b2e9688caddda61"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "8b23aee26e0c67a77df97e659951f469"
server_key_iv: "98076ca9e4060071f1fe49d8afb8e5b4"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.018352 25234 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:09:00.019253 25248 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.019486 25234 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:09:00.019565 25234 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "c6b9cbe7db08476e8b2e9688caddda61"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "8b23aee26e0c67a77df97e659951f469"
server_key_iv: "98076ca9e4060071f1fe49d8afb8e5b4"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.019637 25234 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:09:00.027542 25234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:09:00.027801 25234 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:09:00.027920 25234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:09:00.028133 25234 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:09:00.028473 25234 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:09:00.028512 25234 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.028548 25234 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:09:00.028575 25234 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.035481 25234 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:43381
I20251026 15:09:00.035539 25361 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:43381 every 8 connection(s)
I20251026 15:09:00.035863 25234 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:09:00.038973 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 25234
I20251026 15:09:00.039088 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:09:00.039265 15185 external_mini_cluster.cc:1468] Setting key a10984c844264d8d57d3544fb37bde43
I20251026 15:09:00.040288 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41747
--builtin_ntp_servers=127.14.212.84:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251026 15:09:00.040434 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:00.036659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:59387 (local address 127.14.212.126:41747)
1026 15:09:00.036719 (+ 60us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:00.036721 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:00.036928 (+ 207us) server_negotiation.cc:408] Connection header received
1026 15:09:00.037474 (+ 546us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:00.037476 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:00.037491 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:00.037555 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:00.038010 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:00.038587 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:00.039200 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:00.039377 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:00.040146 (+ 769us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:09:00.040150 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:09:00.040150 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:09:00.040155 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:09:00.040171 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:09:00.040174 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:09:00.040235 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:00.040319 (+ 84us) server_negotiation.cc:300] Negotiation successful
1026 15:09:00.040358 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251026 15:09:00.040647 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:00.036635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:41747 (local address 127.14.212.65:59387)
1026 15:09:00.036812 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:00.036845 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:00.037370 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:00.037616 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:00.037619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:00.037627 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:09:00.037922 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:00.037924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:00.038671 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:09:00.038672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:09:00.039106 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:00.039108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:00.039256 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:00.040028 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:09:00.040028 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:09:00.040055 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:09:00.040074 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:09:00.040240 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:09:00.040244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:09:00.040247 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:09:00.040325 (+ 78us) client_negotiation.cc:240] Negotiation successful
1026 15:09:00.040402 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20251026 15:09:00.041061 25362 heartbeater.cc:344] Connected to a master server at 127.14.212.126:41747
I20251026 15:09:00.041167 25362 heartbeater.cc:461] Registering TS with master...
I20251026 15:09:00.041339 25362 heartbeater.cc:507] Master 127.14.212.126:41747 requested a full tablet report, sending...
I20251026 15:09:00.041851 25174 ts_manager.cc:194] Registered new tserver with Master: c6b9cbe7db08476e8b2e9688caddda61 (127.14.212.65:43381)
I20251026 15:09:00.042958 25174 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.14.212.65:59387
W20251026 15:09:00.122911 25365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:09:00.123106 25365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:09:00.123128 25365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:09:00.124722 25365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:09:00.124778 25365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:09:00.124805 25365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:09:00.126435 25365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:09:00.126659 25365 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:09:00.126921 25365 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:09:00.129626 25371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:00.129654 25373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:00.129703 25365 server_base.cc:1047] running on GCE node
W20251026 15:09:00.129654 25370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:00.129937 25365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:09:00.130136 25365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:09:00.131328 25365 hybrid_clock.cc:648] HybridClock initialized: now 1761491340131312 us; error 35 us; skew 500 ppm
I20251026 15:09:00.132480 25365 webserver.cc:492] Webserver started at http://127.14.212.66:41365/ using document root <none> and password file <none>
I20251026 15:09:00.132678 25365 fs_manager.cc:362] Metadata directory not provided
I20251026 15:09:00.132725 25365 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:09:00.132850 25365 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:09:00.133801 25365 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "eb27c93da2c349fc9e692f4766bf8385"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "9ef1a7cb55fd35fdf2ec83f9a9692dc4"
server_key_iv: "3211120c461b2aed4ab1e45f108b32f9"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.134107 25365 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "eb27c93da2c349fc9e692f4766bf8385"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "9ef1a7cb55fd35fdf2ec83f9a9692dc4"
server_key_iv: "3211120c461b2aed4ab1e45f108b32f9"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.135337 25365 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:09:00.136022 25379 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.136221 25365 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:09:00.136296 25365 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "eb27c93da2c349fc9e692f4766bf8385"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "9ef1a7cb55fd35fdf2ec83f9a9692dc4"
server_key_iv: "3211120c461b2aed4ab1e45f108b32f9"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.136358 25365 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:09:00.171116 25365 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:09:00.171409 25365 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:09:00.171533 25365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:09:00.171767 25365 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:09:00.172087 25365 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:09:00.172120 25365 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.172159 25365 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:09:00.172196 25365 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.177829 25365 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:40851
I20251026 15:09:00.177894 25492 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:40851 every 8 connection(s)
I20251026 15:09:00.178215 25365 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:09:00.182332 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:00.178962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:53599 (local address 127.14.212.126:41747)
1026 15:09:00.179015 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:00.179017 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:00.179227 (+ 210us) server_negotiation.cc:408] Connection header received
1026 15:09:00.179702 (+ 475us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:00.179703 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:00.179716 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:00.179781 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:00.180271 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:00.180751 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:00.181367 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:00.181475 (+ 108us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:00.182085 (+ 610us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:09:00.182088 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:09:00.182089 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:09:00.182093 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:09:00.182102 (+ 9us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:09:00.182104 (+ 2us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:09:00.182146 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:00.182245 (+ 99us) server_negotiation.cc:300] Negotiation successful
1026 15:09:00.182274 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251026 15:09:00.182482 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:00.178934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:41747 (local address 127.14.212.66:53599)
1026 15:09:00.179116 (+ 182us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:00.179148 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:00.179631 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:00.179844 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:00.179847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:00.179857 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:09:00.180150 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:00.180152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:00.180877 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:09:00.180878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:09:00.181302 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:00.181303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:00.181421 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:00.181988 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:09:00.181989 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:09:00.182009 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:09:00.182028 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:09:00.182155 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:09:00.182158 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:09:00.182160 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:09:00.182244 (+ 84us) client_negotiation.cc:240] Negotiation successful
1026 15:09:00.182307 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20251026 15:09:00.183058 25493 heartbeater.cc:344] Connected to a master server at 127.14.212.126:41747
I20251026 15:09:00.183151 25493 heartbeater.cc:461] Registering TS with master...
I20251026 15:09:00.183405 25493 heartbeater.cc:507] Master 127.14.212.126:41747 requested a full tablet report, sending...
I20251026 15:09:00.183790 25174 ts_manager.cc:194] Registered new tserver with Master: eb27c93da2c349fc9e692f4766bf8385 (127.14.212.66:40851)
I20251026 15:09:00.184180 25174 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.14.212.66:53599
I20251026 15:09:00.186030 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 25365
I20251026 15:09:00.186120 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:09:00.186314 15185 external_mini_cluster.cc:1468] Setting key b4db8de17fd71fd7d8c6a9d3834307ee
I20251026 15:09:00.187276 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41747
--builtin_ntp_servers=127.14.212.84:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20251026 15:09:00.268337 25496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:09:00.268535 25496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:09:00.268558 25496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:09:00.270043 25496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:09:00.270100 25496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:09:00.270125 25496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:09:00.271751 25496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:09:00.271942 25496 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:09:00.272147 25496 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:09:00.275149 25501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:00.275148 25504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:00.275288 25502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:00.275311 25496 server_base.cc:1047] running on GCE node
I20251026 15:09:00.275592 25496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:09:00.275815 25496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:09:00.276957 25496 hybrid_clock.cc:648] HybridClock initialized: now 1761491340276941 us; error 28 us; skew 500 ppm
I20251026 15:09:00.278262 25496 webserver.cc:492] Webserver started at http://127.14.212.67:40401/ using document root <none> and password file <none>
I20251026 15:09:00.278501 25496 fs_manager.cc:362] Metadata directory not provided
I20251026 15:09:00.278553 25496 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:09:00.278671 25496 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:09:00.279685 25496 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "a96a388babc94be096f0524c3a426e89"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "e4d39eeb446970ae04f99184baade6f8"
server_key_iv: "b00fbda0c48369f038c24ba46a2af7d0"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.279994 25496 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "a96a388babc94be096f0524c3a426e89"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "e4d39eeb446970ae04f99184baade6f8"
server_key_iv: "b00fbda0c48369f038c24ba46a2af7d0"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.281277 25496 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:09:00.282203 25510 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.282466 25496 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:09:00.282550 25496 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "a96a388babc94be096f0524c3a426e89"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "e4d39eeb446970ae04f99184baade6f8"
server_key_iv: "b00fbda0c48369f038c24ba46a2af7d0"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.282618 25496 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:09:00.319298 25496 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:09:00.319566 25496 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:09:00.319685 25496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:09:00.319907 25496 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:09:00.320235 25496 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:09:00.320279 25496 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.320313 25496 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:09:00.320343 25496 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.325889 25496 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:37405
I20251026 15:09:00.325977 25623 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:37405 every 8 connection(s)
I20251026 15:09:00.326249 25496 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:09:00.330736 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:00.327044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:58475 (local address 127.14.212.126:41747)
1026 15:09:00.327092 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:00.327094 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:00.327366 (+ 272us) server_negotiation.cc:408] Connection header received
1026 15:09:00.327832 (+ 466us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:00.327833 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:00.327843 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:00.327907 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:00.328522 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:00.328994 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:00.329610 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:00.329716 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:00.330369 (+ 653us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:09:00.330374 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:09:00.330375 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:09:00.330380 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:09:00.330428 (+ 48us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:09:00.330433 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:09:00.330522 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:00.330624 (+ 102us) server_negotiation.cc:300] Negotiation successful
1026 15:09:00.330659 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251026 15:09:00.330878 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:00.327015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:41747 (local address 127.14.212.67:58475)
1026 15:09:00.327256 (+ 241us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:00.327282 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:00.327757 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:00.327964 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:00.327967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:00.327976 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:09:00.328404 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:00.328409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:00.329104 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:09:00.329105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:09:00.329531 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:00.329533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:00.329656 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:00.330228 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:09:00.330229 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:09:00.330249 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:09:00.330267 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:09:00.330535 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:09:00.330542 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:09:00.330546 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:09:00.330624 (+ 78us) client_negotiation.cc:240] Negotiation successful
1026 15:09:00.330705 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20251026 15:09:00.331431 25624 heartbeater.cc:344] Connected to a master server at 127.14.212.126:41747
I20251026 15:09:00.331521 25624 heartbeater.cc:461] Registering TS with master...
I20251026 15:09:00.331703 25624 heartbeater.cc:507] Master 127.14.212.126:41747 requested a full tablet report, sending...
I20251026 15:09:00.331761 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 25496
I20251026 15:09:00.331861 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:09:00.332043 15185 external_mini_cluster.cc:1468] Setting key cef9b4c16e435a842ed3bbae9087ccd2
I20251026 15:09:00.332073 25174 ts_manager.cc:194] Registered new tserver with Master: a96a388babc94be096f0524c3a426e89 (127.14.212.67:37405)
I20251026 15:09:00.332533 25174 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.14.212.67:58475
I20251026 15:09:00.333030 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251026 15:09:00.334897 25214 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:09:00.334359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.14.212.126:41747)
1026 15:09:00.334438 (+ 79us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:00.334440 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:00.334569 (+ 129us) server_negotiation.cc:408] Connection header received
1026 15:09:00.334636 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:00.334637 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:00.334647 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:00.334698 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:00.334837 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42714: BlockingRecv error: recv got EOF from 127.0.0.1:42714 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20251026 15:09:00.335021 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 25234
I20251026 15:09:00.340044 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 25365
I20251026 15:09:00.345630 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 25496
I20251026 15:09:00.350067 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 25143
2025-10-26T15:09:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (608 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-10-26T15:09:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:09:00Z Disabled control of system clock
I20251026 15:09:00.376394 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:42111
--webserver_interface=127.14.212.126
--webserver_port=0
--builtin_ntp_servers=127.14.212.84:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:42111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251026 15:09:00.459432 25635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:09:00.459636 25635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:09:00.459659 25635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:09:00.461206 25635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:09:00.461273 25635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:09:00.461287 25635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:09:00.461305 25635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:09:00.461318 25635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:09:00.463038 25635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:42111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:42111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:09:00.463292 25635 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:09:00.463552 25635 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:09:00.466305 25640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:00.466322 25641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:00.466475 25635 server_base.cc:1047] running on GCE node
W20251026 15:09:00.466360 25643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:00.466763 25635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:09:00.466979 25635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:09:00.468104 25635 hybrid_clock.cc:648] HybridClock initialized: now 1761491340468090 us; error 38 us; skew 500 ppm
I20251026 15:09:00.469292 25635 webserver.cc:492] Webserver started at http://127.14.212.126:37175/ using document root <none> and password file <none>
I20251026 15:09:00.469487 25635 fs_manager.cc:362] Metadata directory not provided
I20251026 15:09:00.469537 25635 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:09:00.469655 25635 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:09:00.470728 25635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "1a2f406faa61428ab325290513cb81a5"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "a97ebdd18c04657e33f1f5c46dfc7eea"
server_key_iv: "2f1c3b23d9f014c1fcb1eb3374a59d90"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.471037 25635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "1a2f406faa61428ab325290513cb81a5"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "a97ebdd18c04657e33f1f5c46dfc7eea"
server_key_iv: "2f1c3b23d9f014c1fcb1eb3374a59d90"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.472299 25635 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:09:00.473107 25649 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.473346 25635 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:09:00.473423 25635 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "1a2f406faa61428ab325290513cb81a5"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "a97ebdd18c04657e33f1f5c46dfc7eea"
server_key_iv: "2f1c3b23d9f014c1fcb1eb3374a59d90"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.473488 25635 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:09:00.476996 25635 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:09:00.477239 25635 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:09:00.477341 25635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:09:00.481223 25635 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:42111
I20251026 15:09:00.481283 25701 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:42111 every 8 connection(s)
I20251026 15:09:00.481583 25635 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:09:00.482249 25702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:09:00.484746 25702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5: Bootstrap starting.
I20251026 15:09:00.485347 25702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5: Neither blocks nor log segments found. Creating new log.
I20251026 15:09:00.485590 25702 log.cc:826] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5: Log is configured to *not* fsync() on all Append() calls
I20251026 15:09:00.486331 25702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5: No bootstrap required, opened a new log
I20251026 15:09:00.487556 25702 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2f406faa61428ab325290513cb81a5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42111 } }
I20251026 15:09:00.487679 25702 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:09:00.487700 25702 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a2f406faa61428ab325290513cb81a5, State: Initialized, Role: FOLLOWER
I20251026 15:09:00.487811 25702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2f406faa61428ab325290513cb81a5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42111 } }
I20251026 15:09:00.487874 25702 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:09:00.487900 25702 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:09:00.487941 25702 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:09:00.488474 25702 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2f406faa61428ab325290513cb81a5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42111 } }
I20251026 15:09:00.488572 25702 leader_election.cc:304] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a2f406faa61428ab325290513cb81a5; no voters:
I20251026 15:09:00.488732 25702 leader_election.cc:290] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:09:00.488782 25705 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:09:00.488931 25705 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [term 1 LEADER]: Becoming Leader. State: Replica: 1a2f406faa61428ab325290513cb81a5, State: Running, Role: LEADER
I20251026 15:09:00.488968 25702 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:09:00.489018 25705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2f406faa61428ab325290513cb81a5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42111 } }
I20251026 15:09:00.489359 25706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a2f406faa61428ab325290513cb81a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2f406faa61428ab325290513cb81a5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42111 } } }
I20251026 15:09:00.489506 25706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [sys.catalog]: This master's current role is: LEADER
I20251026 15:09:00.489382 25707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a2f406faa61428ab325290513cb81a5. Latest consensus state: current_term: 1 leader_uuid: "1a2f406faa61428ab325290513cb81a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2f406faa61428ab325290513cb81a5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 42111 } } }
I20251026 15:09:00.489632 25707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5 [sys.catalog]: This master's current role is: LEADER
I20251026 15:09:00.490011 25712 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:09:00.490461 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 25635
I20251026 15:09:00.490523 25712 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:09:00.490566 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:09:00.490762 15185 external_mini_cluster.cc:1468] Setting key 835497fba62e4f5419dbdfee47d654c0
I20251026 15:09:00.492074 25712 catalog_manager.cc:1357] Generated new cluster ID: 3f9f1e6ce8f9480493d3742869d7c9b3
I20251026 15:09:00.492131 25712 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:09:00.493633 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:00.491409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.14.212.126:42111)
1026 15:09:00.492169 (+ 760us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:00.492175 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:00.492198 (+ 23us) server_negotiation.cc:408] Connection header received
1026 15:09:00.492309 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:00.492314 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:00.492367 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:00.492492 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:00.493228 (+ 736us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:09:00.493231 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:09:00.493232 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:09:00.493236 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:09:00.493274 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:09:00.493280 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:09:00.493361 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:00.493444 (+ 83us) server_negotiation.cc:300] Negotiation successful
1026 15:09:00.493489 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":496,"threads_started":1}
I20251026 15:09:00.498749 25712 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:09:00.498899 25712 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:09:00.508081 25712 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 1a2f406faa61428ab325290513cb81a5: Generated new TSK 0
I20251026 15:09:00.508339 25712 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:09:00.511531 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:42111
--builtin_ntp_servers=127.14.212.84:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251026 15:09:00.591915 25726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:09:00.592095 25726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:09:00.592130 25726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:09:00.593564 25726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:09:00.593609 25726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:09:00.593650 25726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:09:00.595194 25726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:42111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:09:00.595427 25726 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:09:00.595678 25726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:09:00.598274 25732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:00.598358 25726 server_base.cc:1047] running on GCE node
W20251026 15:09:00.598310 25734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:00.598510 25731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:00.598685 25726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:09:00.598898 25726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:09:00.600054 25726 hybrid_clock.cc:648] HybridClock initialized: now 1761491340600038 us; error 33 us; skew 500 ppm
I20251026 15:09:00.601223 25726 webserver.cc:492] Webserver started at http://127.14.212.65:37759/ using document root <none> and password file <none>
I20251026 15:09:00.601429 25726 fs_manager.cc:362] Metadata directory not provided
I20251026 15:09:00.601472 25726 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:09:00.601586 25726 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:09:00.602593 25726 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "ce5444479d594345a711af1f04027652"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "57967418a497200de40713060d6fb90a"
server_key_iv: "95f1c9c0778530954bf59028d42aeda8"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.602895 25726 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "ce5444479d594345a711af1f04027652"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "57967418a497200de40713060d6fb90a"
server_key_iv: "95f1c9c0778530954bf59028d42aeda8"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.604061 25726 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:09:00.604842 25740 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.605086 25726 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:09:00.605239 25726 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "ce5444479d594345a711af1f04027652"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "57967418a497200de40713060d6fb90a"
server_key_iv: "95f1c9c0778530954bf59028d42aeda8"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.605340 25726 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:09:00.608970 25726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:09:00.609169 25726 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:09:00.609277 25726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:09:00.609472 25726 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:09:00.609782 25726 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:09:00.609819 25726 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.609850 25726 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:09:00.609879 25726 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.616791 25726 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:39899
I20251026 15:09:00.616850 25853 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:39899 every 8 connection(s)
I20251026 15:09:00.617156 25726 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:09:00.619930 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:00.617874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:58685 (local address 127.14.212.126:42111)
1026 15:09:00.617943 (+ 69us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:00.617947 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:00.618190 (+ 243us) server_negotiation.cc:408] Connection header received
1026 15:09:00.618819 (+ 629us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:00.618820 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:00.618825 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:00.618880 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:00.619705 (+ 825us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:09:00.619706 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:09:00.619707 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:09:00.619711 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:09:00.619726 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:09:00.619729 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:09:00.619770 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:00.619862 (+ 92us) server_negotiation.cc:300] Negotiation successful
1026 15:09:00.619876 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251026 15:09:00.620110 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:00.617860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:42111 (local address 127.14.212.65:58685)
1026 15:09:00.618072 (+ 212us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:00.618103 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:00.618712 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:00.618942 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:00.618946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:00.618956 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:09:00.619561 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:09:00.619562 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:09:00.619589 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:09:00.619613 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:09:00.619784 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:09:00.619788 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:09:00.619790 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:09:00.619880 (+ 90us) client_negotiation.cc:240] Negotiation successful
1026 15:09:00.619932 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20251026 15:09:00.620645 25854 heartbeater.cc:344] Connected to a master server at 127.14.212.126:42111
I20251026 15:09:00.620741 25854 heartbeater.cc:461] Registering TS with master...
I20251026 15:09:00.620868 25854 heartbeater.cc:507] Master 127.14.212.126:42111 requested a full tablet report, sending...
I20251026 15:09:00.621289 25666 ts_manager.cc:194] Registered new tserver with Master: ce5444479d594345a711af1f04027652 (127.14.212.65:39899)
I20251026 15:09:00.626197 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 25726
I20251026 15:09:00.626325 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:09:00.626525 15185 external_mini_cluster.cc:1468] Setting key 7dbc5e328ebd0a27ce2d392c27459320
I20251026 15:09:00.627548 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:42111
--builtin_ntp_servers=127.14.212.84:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251026 15:09:00.707479 25857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:09:00.707669 25857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:09:00.707703 25857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:09:00.709296 25857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:09:00.709352 25857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:09:00.709390 25857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:09:00.710997 25857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:42111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:09:00.711244 25857 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:09:00.711479 25857 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:09:00.714236 25862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:00.714274 25857 server_base.cc:1047] running on GCE node
W20251026 15:09:00.714191 25863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:00.714254 25865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:00.714668 25857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:09:00.714884 25857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:09:00.716028 25857 hybrid_clock.cc:648] HybridClock initialized: now 1761491340716015 us; error 31 us; skew 500 ppm
I20251026 15:09:00.717293 25857 webserver.cc:492] Webserver started at http://127.14.212.66:35569/ using document root <none> and password file <none>
I20251026 15:09:00.717525 25857 fs_manager.cc:362] Metadata directory not provided
I20251026 15:09:00.717573 25857 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:09:00.717691 25857 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:09:00.718854 25857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "a0b0de89b0d04e879b0c4d44b89750f8"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "d5c73a4deb0cacd70dc60c839720bcdc"
server_key_iv: "b2683f8c506788708eb14a775e8e8267"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.719182 25857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "a0b0de89b0d04e879b0c4d44b89750f8"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "d5c73a4deb0cacd70dc60c839720bcdc"
server_key_iv: "b2683f8c506788708eb14a775e8e8267"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.720403 25857 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:09:00.721175 25871 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.721432 25857 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:09:00.721522 25857 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "a0b0de89b0d04e879b0c4d44b89750f8"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "d5c73a4deb0cacd70dc60c839720bcdc"
server_key_iv: "b2683f8c506788708eb14a775e8e8267"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.721586 25857 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:09:00.725080 25857 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:09:00.725289 25857 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:09:00.725396 25857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:09:00.725571 25857 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:09:00.725876 25857 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:09:00.725908 25857 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.725943 25857 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:09:00.725975 25857 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.731758 25857 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:32807
I20251026 15:09:00.731849 25984 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:32807 every 8 connection(s)
I20251026 15:09:00.732172 25857 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:09:00.734800 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:00.732885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:38657 (local address 127.14.212.126:42111)
1026 15:09:00.732954 (+ 69us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:00.732957 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:00.733133 (+ 176us) server_negotiation.cc:408] Connection header received
1026 15:09:00.733615 (+ 482us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:00.733616 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:00.733625 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:00.733668 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:00.734523 (+ 855us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:09:00.734527 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:09:00.734527 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:09:00.734532 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:09:00.734550 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:09:00.734554 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:09:00.734619 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:00.734724 (+ 105us) server_negotiation.cc:300] Negotiation successful
1026 15:09:00.734742 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251026 15:09:00.734906 25987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:00.732853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:42111 (local address 127.14.212.66:38657)
1026 15:09:00.733026 (+ 173us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:00.733051 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:00.733525 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:00.733732 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:00.733735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:00.733744 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:09:00.734325 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:09:00.734325 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:09:00.734346 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:09:00.734366 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:09:00.734622 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:09:00.734625 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:09:00.734627 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:09:00.734703 (+ 76us) client_negotiation.cc:240] Negotiation successful
1026 15:09:00.734747 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20251026 15:09:00.735383 25985 heartbeater.cc:344] Connected to a master server at 127.14.212.126:42111
I20251026 15:09:00.735467 25985 heartbeater.cc:461] Registering TS with master...
I20251026 15:09:00.735594 25985 heartbeater.cc:507] Master 127.14.212.126:42111 requested a full tablet report, sending...
I20251026 15:09:00.735908 25666 ts_manager.cc:194] Registered new tserver with Master: a0b0de89b0d04e879b0c4d44b89750f8 (127.14.212.66:32807)
I20251026 15:09:00.742136 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 25857
I20251026 15:09:00.742252 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:09:00.742486 15185 external_mini_cluster.cc:1468] Setting key ffed1067c12686fd27ec26a9bd0a96f6
I20251026 15:09:00.743643 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:42111
--builtin_ntp_servers=127.14.212.84:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251026 15:09:00.823462 25988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:09:00.823664 25988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:09:00.823702 25988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:09:00.825294 25988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:09:00.825348 25988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:09:00.825389 25988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:09:00.827019 25988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:42111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:09:00.827272 25988 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:09:00.827513 25988 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:09:00.830183 25993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:00.830184 25996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:00.830371 25988 server_base.cc:1047] running on GCE node
W20251026 15:09:00.830252 25994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:00.830688 25988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:09:00.830873 25988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:09:00.832000 25988 hybrid_clock.cc:648] HybridClock initialized: now 1761491340831982 us; error 36 us; skew 500 ppm
I20251026 15:09:00.833197 25988 webserver.cc:492] Webserver started at http://127.14.212.67:37043/ using document root <none> and password file <none>
I20251026 15:09:00.833431 25988 fs_manager.cc:362] Metadata directory not provided
I20251026 15:09:00.833477 25988 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:09:00.833585 25988 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:09:00.834604 25988 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "d839b0fb00a14509930651331e36b16c"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "f5140cfc355e1b1e6c38aae938714828"
server_key_iv: "45b60a598433be714ac9808cfcc46f38"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.834913 25988 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "d839b0fb00a14509930651331e36b16c"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "f5140cfc355e1b1e6c38aae938714828"
server_key_iv: "45b60a598433be714ac9808cfcc46f38"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.836311 25988 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:09:00.837306 26002 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.837533 25988 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:09:00.837625 25988 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "d839b0fb00a14509930651331e36b16c"
format_stamp: "Formatted at 2025-10-26 15:09:00 on dist-test-slave-8pgz"
server_key: "f5140cfc355e1b1e6c38aae938714828"
server_key_iv: "45b60a598433be714ac9808cfcc46f38"
server_key_version: "encryptionkey@0"
I20251026 15:09:00.837693 25988 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:09:00.841259 25988 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:09:00.841476 25988 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:09:00.841583 25988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:09:00.841744 25988 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:09:00.842016 25988 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:09:00.842049 25988 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.842082 25988 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:09:00.842115 25988 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:00.847769 25988 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:40455
I20251026 15:09:00.847816 26115 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:40455 every 8 connection(s)
I20251026 15:09:00.848111 25988 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:09:00.850646 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:00.848830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:53977 (local address 127.14.212.126:42111)
1026 15:09:00.848912 (+ 82us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:00.848914 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:00.849038 (+ 124us) server_negotiation.cc:408] Connection header received
1026 15:09:00.849586 (+ 548us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:00.849587 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:00.849593 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:00.849666 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:00.850360 (+ 694us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:09:00.850362 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:09:00.850362 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:09:00.850366 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:09:00.850382 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:09:00.850386 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:09:00.850467 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:00.850554 (+ 87us) server_negotiation.cc:300] Negotiation successful
1026 15:09:00.850572 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251026 15:09:00.850802 26118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:00.848743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:42111 (local address 127.14.212.67:53977)
1026 15:09:00.848934 (+ 191us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:00.848961 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:00.849484 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:00.849724 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:00.849728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:00.849736 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:09:00.850249 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:09:00.850250 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:09:00.850266 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:09:00.850284 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:09:00.850472 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:09:00.850476 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:09:00.850479 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:09:00.850576 (+ 97us) client_negotiation.cc:240] Negotiation successful
1026 15:09:00.850627 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20251026 15:09:00.851289 26116 heartbeater.cc:344] Connected to a master server at 127.14.212.126:42111
I20251026 15:09:00.851366 26116 heartbeater.cc:461] Registering TS with master...
I20251026 15:09:00.851491 26116 heartbeater.cc:507] Master 127.14.212.126:42111 requested a full tablet report, sending...
I20251026 15:09:00.851861 25666 ts_manager.cc:194] Registered new tserver with Master: d839b0fb00a14509930651331e36b16c (127.14.212.67:40455)
I20251026 15:09:00.858330 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 25988
I20251026 15:09:00.858490 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:09:00.858683 15185 external_mini_cluster.cc:1468] Setting key df3e26d61f743134461280c3125b6202
I20251026 15:09:00.859580 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251026 15:09:00.861402 25725 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:09:00.860938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.14.212.126:42111)
1026 15:09:00.860997 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:00.860999 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:00.861110 (+ 111us) server_negotiation.cc:408] Connection header received
1026 15:09:00.861138 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:00.861139 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:00.861147 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:00.861198 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:00.861330 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34454: BlockingRecv error: recv got EOF from 127.0.0.1:34454 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20251026 15:09:00.861536 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 25726
I20251026 15:09:00.866041 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 25857
I20251026 15:09:00.871672 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 25988
I20251026 15:09:00.875891 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 25635
2025-10-26T15:09:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (525 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:09:00 dist-test-slave-8pgz krb5kdc[26126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:09:00 dist-test-slave-8pgz krb5kdc[26126](info): set up 2 sockets
Oct 26 15:09:00 dist-test-slave-8pgz krb5kdc[26126](info): commencing operation
krb5kdc: starting...
W20251026 15:09:02.864329 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.969s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:09:02 dist-test-slave-8pgz krb5kdc[26126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:09:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:09:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:09:03.001297 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:40829
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:40829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:09:03.079566 26142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:09:03.079756 26142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:09:03.079792 26142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:09:03.081211 26142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:09:03.081264 26142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:09:03.081287 26142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:09:03.081306 26142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:09:03.081324 26142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:09:03.082928 26142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:40829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:40829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:09:03.083165 26142 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:09:03.083344 26142 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:09:03.085851 26150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:03.085858 26147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:03.086064 26142 server_base.cc:1047] running on GCE node
W20251026 15:09:03.085866 26148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:03.086416 26142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:09:03.086658 26142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:09:03.087813 26142 hybrid_clock.cc:648] HybridClock initialized: now 1761491343087808 us; error 30 us; skew 500 ppm
Oct 26 15:09:03 dist-test-slave-8pgz krb5kdc[26126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:09:03.090081 26142 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:09:03.090656 26142 webserver.cc:492] Webserver started at http://127.14.212.126:38315/ using document root <none> and password file <none>
I20251026 15:09:03.090826 26142 fs_manager.cc:362] Metadata directory not provided
I20251026 15:09:03.090857 26142 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:09:03.090937 26142 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:09:03.091836 26142 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "97ef8bbb72374146a3012ce787c01f38"
format_stamp: "Formatted at 2025-10-26 15:09:03 on dist-test-slave-8pgz"
server_key: "da4d162ecd231cc47b8da35301585b8b"
server_key_iv: "28cf99e5bae718d757ba7bbdf112b193"
server_key_version: "encryptionkey@0"
I20251026 15:09:03.092100 26142 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "97ef8bbb72374146a3012ce787c01f38"
format_stamp: "Formatted at 2025-10-26 15:09:03 on dist-test-slave-8pgz"
server_key: "da4d162ecd231cc47b8da35301585b8b"
server_key_iv: "28cf99e5bae718d757ba7bbdf112b193"
server_key_version: "encryptionkey@0"
I20251026 15:09:03.093215 26142 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:09:03.093855 26157 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:03.094023 26142 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:09:03.094091 26142 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "97ef8bbb72374146a3012ce787c01f38"
format_stamp: "Formatted at 2025-10-26 15:09:03 on dist-test-slave-8pgz"
server_key: "da4d162ecd231cc47b8da35301585b8b"
server_key_iv: "28cf99e5bae718d757ba7bbdf112b193"
server_key_version: "encryptionkey@0"
I20251026 15:09:03.094141 26142 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:09:03.133636 26142 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:09:03.139493 26142 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:09:03.139606 26142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:09:03.143433 26142 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:40829
I20251026 15:09:03.143465 26209 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:40829 every 8 connection(s)
I20251026 15:09:03.143783 26142 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:09:03.144399 26210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:09:03.145969 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 26142
I20251026 15:09:03.146078 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:09:03.146270 15185 external_mini_cluster.cc:1468] Setting key f0673c04e70936ee51a789792b7271a1
I20251026 15:09:03.147137 26210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38: Bootstrap starting.
I20251026 15:09:03.147773 26210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38: Neither blocks nor log segments found. Creating new log.
I20251026 15:09:03.148064 26210 log.cc:826] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38: Log is configured to *not* fsync() on all Append() calls
I20251026 15:09:03.148798 26210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38: No bootstrap required, opened a new log
I20251026 15:09:03.150146 26210 raft_consensus.cc:359] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ef8bbb72374146a3012ce787c01f38" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 40829 } }
I20251026 15:09:03.150297 26210 raft_consensus.cc:385] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:09:03.150332 26210 raft_consensus.cc:740] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ef8bbb72374146a3012ce787c01f38, State: Initialized, Role: FOLLOWER
I20251026 15:09:03.150470 26210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ef8bbb72374146a3012ce787c01f38" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 40829 } }
I20251026 15:09:03.150549 26210 raft_consensus.cc:399] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:09:03.150580 26210 raft_consensus.cc:493] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:09:03.150630 26210 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:09:03.151497 26210 raft_consensus.cc:515] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ef8bbb72374146a3012ce787c01f38" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 40829 } }
I20251026 15:09:03.151631 26210 leader_election.cc:304] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97ef8bbb72374146a3012ce787c01f38; no voters:
I20251026 15:09:03.151803 26210 leader_election.cc:290] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:09:03.151860 26215 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:09:03.152112 26215 raft_consensus.cc:697] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [term 1 LEADER]: Becoming Leader. State: Replica: 97ef8bbb72374146a3012ce787c01f38, State: Running, Role: LEADER
I20251026 15:09:03.152179 26210 sys_catalog.cc:565] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [sys.catalog]: configured and running, proceeding with master startup.
Oct 26 15:09:03 dist-test-slave-8pgz krb5kdc[26126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:09:03.152907 26215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ef8bbb72374146a3012ce787c01f38" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 40829 } }
I20251026 15:09:03.153386 26217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97ef8bbb72374146a3012ce787c01f38. Latest consensus state: current_term: 1 leader_uuid: "97ef8bbb72374146a3012ce787c01f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ef8bbb72374146a3012ce787c01f38" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 40829 } } }
I20251026 15:09:03.153764 26216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97ef8bbb72374146a3012ce787c01f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ef8bbb72374146a3012ce787c01f38" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 40829 } } }
I20251026 15:09:03.153882 26216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [sys.catalog]: This master's current role is: LEADER
I20251026 15:09:03.153553 26217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38 [sys.catalog]: This master's current role is: LEADER
I20251026 15:09:03.154196 26226 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:09:03.155426 26226 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:09:03.156988 26226 catalog_manager.cc:1357] Generated new cluster ID: 1ed90476bfad45b5b246947d6fd2f3b8
I20251026 15:09:03.157032 26226 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:09:03.157199 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.146856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.14.212.126:40829)
1026 15:09:03.147005 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:03.147014 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:03.147031 (+ 17us) server_negotiation.cc:408] Connection header received
1026 15:09:03.147373 (+ 342us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:03.147380 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:03.147427 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:03.147547 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:03.148511 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.149256 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.150010 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.150177 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.152953 (+ 2776us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:09:03.152962 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:09:03.152962 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:09:03.152989 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:09:03.155364 (+ 2375us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:09:03.155768 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:09:03.155771 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:09:03.155771 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:09:03.155834 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:09:03.155996 (+ 162us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:09:03.155996 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:09:03.155997 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:09:03.156248 (+ 251us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:09:03.156395 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:03.156892 (+ 497us) server_negotiation.cc:300] Negotiation successful
1026 15:09:03.157035 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20251026 15:09:03.180840 26226 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:09:03.181494 26226 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:09:03.192956 26226 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38: Generated new TSK 0
I20251026 15:09:03.193179 26226 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:09:03.249929 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:40829
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:09:03.353996 26238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:09:03.354234 26238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:09:03.354274 26238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:09:03.355806 26238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:09:03.355860 26238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:09:03.355903 26238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:09:03.357654 26238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:40829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:09:03.357915 26238 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:09:03.358129 26238 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:09:03.361054 26243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:03.361056 26244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:03.361059 26246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:03.361338 26238 server_base.cc:1047] running on GCE node
I20251026 15:09:03.361522 26238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:09:03.361730 26238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:09:03.362901 26238 hybrid_clock.cc:648] HybridClock initialized: now 1761491343362886 us; error 34 us; skew 500 ppm
Oct 26 15:09:03 dist-test-slave-8pgz krb5kdc[26126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:09:03.365442 26238 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:09:03.365998 26238 webserver.cc:492] Webserver started at http://127.14.212.65:42841/ using document root <none> and password file <none>
I20251026 15:09:03.366195 26238 fs_manager.cc:362] Metadata directory not provided
I20251026 15:09:03.366235 26238 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:09:03.366350 26238 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:09:03.367381 26238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "f1b041cb538f437dac8cfa44cb717dd1"
format_stamp: "Formatted at 2025-10-26 15:09:03 on dist-test-slave-8pgz"
server_key: "3e401d87c51b12ed27211ae43c0c9c27"
server_key_iv: "ef9c77ecbe877b620363491033b95fab"
server_key_version: "encryptionkey@0"
I20251026 15:09:03.367702 26238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "f1b041cb538f437dac8cfa44cb717dd1"
format_stamp: "Formatted at 2025-10-26 15:09:03 on dist-test-slave-8pgz"
server_key: "3e401d87c51b12ed27211ae43c0c9c27"
server_key_iv: "ef9c77ecbe877b620363491033b95fab"
server_key_version: "encryptionkey@0"
I20251026 15:09:03.368904 26238 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:09:03.369625 26253 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:03.369809 26238 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:09:03.369875 26238 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "f1b041cb538f437dac8cfa44cb717dd1"
format_stamp: "Formatted at 2025-10-26 15:09:03 on dist-test-slave-8pgz"
server_key: "3e401d87c51b12ed27211ae43c0c9c27"
server_key_iv: "ef9c77ecbe877b620363491033b95fab"
server_key_version: "encryptionkey@0"
I20251026 15:09:03.369939 26238 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:09:03.390694 26238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:09:03.392760 26238 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:09:03.392923 26238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:09:03.393164 26238 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:09:03.393498 26238 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:09:03.393534 26238 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:03.393570 26238 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:09:03.393603 26238 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:03.399588 26238 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:42937
I20251026 15:09:03.399668 26366 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:42937 every 8 connection(s)
I20251026 15:09:03.399964 26238 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
Oct 26 15:09:03 dist-test-slave-8pgz krb5kdc[26126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:09:03.407509 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 26238
I20251026 15:09:03.407629 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:09:03.407842 15185 external_mini_cluster.cc:1468] Setting key 146a37adef3138c70d0b30ce1626b60d
I20251026 15:09:03.409114 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.400692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:42217 (local address 127.14.212.126:40829)
1026 15:09:03.400768 (+ 76us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:03.400770 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:03.400900 (+ 130us) server_negotiation.cc:408] Connection header received
1026 15:09:03.401420 (+ 520us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:03.401422 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:03.401434 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:03.401491 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:03.402277 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.402847 (+ 570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.403402 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.403576 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.406112 (+ 2536us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:09:03.406119 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:09:03.406119 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:09:03.406127 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:09:03.407776 (+ 1649us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:09:03.408205 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:09:03.408207 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:09:03.408207 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:09:03.408238 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:09:03.408410 (+ 172us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:09:03.408411 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:09:03.408411 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:09:03.408520 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:09:03.408600 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:03.408956 (+ 356us) server_negotiation.cc:300] Negotiation successful
1026 15:09:03.409048 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251026 15:09:03.409375 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.400628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:40829 (local address 127.14.212.65:42217)
1026 15:09:03.400796 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:03.400823 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:03.401321 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:03.401577 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:03.401580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:03.401894 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:09:03.402174 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.402176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.402938 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.402940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:09:03.403316 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.403318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.403459 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.404568 (+ 1109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:09:03.404584 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:09:03.406026 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:09:03.407860 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:09:03.407866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:09:03.407869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:09:03.408109 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:09:03.408298 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:09:03.408299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:09:03.408299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:09:03.408360 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:09:03.408590 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:09:03.408596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:09:03.408770 (+ 174us) client_negotiation.cc:769] Sending connection context
1026 15:09:03.408901 (+ 131us) client_negotiation.cc:240] Negotiation successful
1026 15:09:03.409061 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20251026 15:09:03.409914 26367 heartbeater.cc:344] Connected to a master server at 127.14.212.126:40829
I20251026 15:09:03.410096 26367 heartbeater.cc:461] Registering TS with master...
I20251026 15:09:03.410425 26367 heartbeater.cc:507] Master 127.14.212.126:40829 requested a full tablet report, sending...
I20251026 15:09:03.410991 26174 ts_manager.cc:194] Registered new tserver with Master: f1b041cb538f437dac8cfa44cb717dd1 (127.14.212.65:42937)
I20251026 15:09:03.411681 26174 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:42217
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:09:03.456969 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:40829
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:09:03.539074 26374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:09:03.539287 26374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:09:03.539330 26374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:09:03.540808 26374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:09:03.540858 26374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:09:03.540902 26374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:09:03.542632 26374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:40829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:09:03.542887 26374 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:09:03.543087 26374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:09:03.545801 26380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:03.545796 26382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:03.545818 26379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:03.545938 26374 server_base.cc:1047] running on GCE node
I20251026 15:09:03.546166 26374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:09:03.546382 26374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:09:03.547586 26374 hybrid_clock.cc:648] HybridClock initialized: now 1761491343547575 us; error 42 us; skew 500 ppm
Oct 26 15:09:03 dist-test-slave-8pgz krb5kdc[26126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:09:03.550271 26374 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:09:03.550887 26374 webserver.cc:492] Webserver started at http://127.14.212.66:46547/ using document root <none> and password file <none>
I20251026 15:09:03.551100 26374 fs_manager.cc:362] Metadata directory not provided
I20251026 15:09:03.551141 26374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:09:03.551270 26374 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:09:03.552284 26374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "58273c39e8674f2b91747ec44b07badd"
format_stamp: "Formatted at 2025-10-26 15:09:03 on dist-test-slave-8pgz"
server_key: "7db33bd0715df1f9af26fe63fb172f30"
server_key_iv: "c7f596f9e71705735285bb8eb3f86862"
server_key_version: "encryptionkey@0"
I20251026 15:09:03.552613 26374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "58273c39e8674f2b91747ec44b07badd"
format_stamp: "Formatted at 2025-10-26 15:09:03 on dist-test-slave-8pgz"
server_key: "7db33bd0715df1f9af26fe63fb172f30"
server_key_iv: "c7f596f9e71705735285bb8eb3f86862"
server_key_version: "encryptionkey@0"
I20251026 15:09:03.553998 26374 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:09:03.554795 26389 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:03.554982 26374 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251026 15:09:03.555056 26374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "58273c39e8674f2b91747ec44b07badd"
format_stamp: "Formatted at 2025-10-26 15:09:03 on dist-test-slave-8pgz"
server_key: "7db33bd0715df1f9af26fe63fb172f30"
server_key_iv: "c7f596f9e71705735285bb8eb3f86862"
server_key_version: "encryptionkey@0"
I20251026 15:09:03.555121 26374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:09:03.566381 26374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:09:03.568286 26374 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:09:03.568421 26374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:09:03.568662 26374 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:09:03.568964 26374 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:09:03.568995 26374 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:03.569031 26374 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:09:03.569062 26374 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:03.575333 26374 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:36425
I20251026 15:09:03.575429 26502 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:36425 every 8 connection(s)
I20251026 15:09:03.575699 26374 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
Oct 26 15:09:03 dist-test-slave-8pgz krb5kdc[26126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:09:03.581583 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 26374
I20251026 15:09:03.581700 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:09:03.581887 15185 external_mini_cluster.cc:1468] Setting key 579911fa5b77dbd3850cd449d13d051a
I20251026 15:09:03.584530 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.576438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:58687 (local address 127.14.212.126:40829)
1026 15:09:03.576514 (+ 76us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:03.576517 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:03.576637 (+ 120us) server_negotiation.cc:408] Connection header received
1026 15:09:03.577140 (+ 503us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:03.577141 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:03.577153 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:03.577208 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:03.577999 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.578584 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.579129 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.579310 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.581666 (+ 2356us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:09:03.581671 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:09:03.581672 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:09:03.581680 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:09:03.583217 (+ 1537us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:09:03.583694 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:09:03.583695 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:09:03.583695 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:09:03.583715 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:09:03.583916 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:09:03.583917 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:09:03.583917 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:09:03.584006 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:09:03.584068 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:03.584343 (+ 275us) server_negotiation.cc:300] Negotiation successful
1026 15:09:03.584436 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251026 15:09:03.584676 26505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.576342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:40829 (local address 127.14.212.66:58687)
1026 15:09:03.576507 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:03.576549 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:03.577051 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:03.577269 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:03.577273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:03.577613 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:09:03.577897 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.577900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.578674 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.578675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:09:03.579053 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.579054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.579189 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.580139 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:09:03.580155 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:09:03.581564 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:09:03.583298 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:09:03.583305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:09:03.583308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:09:03.583619 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:09:03.583785 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:09:03.583787 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:09:03.583787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:09:03.583852 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:09:03.584062 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:09:03.584066 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:09:03.584193 (+ 127us) client_negotiation.cc:769] Sending connection context
1026 15:09:03.584317 (+ 124us) client_negotiation.cc:240] Negotiation successful
1026 15:09:03.584462 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20251026 15:09:03.585127 26503 heartbeater.cc:344] Connected to a master server at 127.14.212.126:40829
I20251026 15:09:03.585237 26503 heartbeater.cc:461] Registering TS with master...
I20251026 15:09:03.585484 26503 heartbeater.cc:507] Master 127.14.212.126:40829 requested a full tablet report, sending...
I20251026 15:09:03.585868 26174 ts_manager.cc:194] Registered new tserver with Master: 58273c39e8674f2b91747ec44b07badd (127.14.212.66:36425)
I20251026 15:09:03.586241 26174 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:58687
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:09:03.629848 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:40829
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:09:03.714027 26510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:09:03.714223 26510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:09:03.714254 26510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:09:03.715723 26510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:09:03.715778 26510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:09:03.715806 26510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:09:03.717396 26510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:40829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:09:03.717583 26510 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:09:03.717734 26510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:09:03.720726 26516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:03.720734 26510 server_base.cc:1047] running on GCE node
W20251026 15:09:03.720717 26515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:09:03.720714 26518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:09:03.721194 26510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:09:03.721446 26510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:09:03.722591 26510 hybrid_clock.cc:648] HybridClock initialized: now 1761491343722578 us; error 29 us; skew 500 ppm
Oct 26 15:09:03 dist-test-slave-8pgz krb5kdc[26126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:09:03.725176 26510 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:09:03.725852 26510 webserver.cc:492] Webserver started at http://127.14.212.67:46477/ using document root <none> and password file <none>
I20251026 15:09:03.726064 26510 fs_manager.cc:362] Metadata directory not provided
I20251026 15:09:03.726109 26510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:09:03.726220 26510 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:09:03.727221 26510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "2af81dc46d90481cb4c9634019878ea4"
format_stamp: "Formatted at 2025-10-26 15:09:03 on dist-test-slave-8pgz"
server_key: "c5524bad71b0deb2ddf13bcd80932223"
server_key_iv: "f061b1d18af56d4d110602ccae5bec2a"
server_key_version: "encryptionkey@0"
I20251026 15:09:03.727562 26510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "2af81dc46d90481cb4c9634019878ea4"
format_stamp: "Formatted at 2025-10-26 15:09:03 on dist-test-slave-8pgz"
server_key: "c5524bad71b0deb2ddf13bcd80932223"
server_key_iv: "f061b1d18af56d4d110602ccae5bec2a"
server_key_version: "encryptionkey@0"
I20251026 15:09:03.728812 26510 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:09:03.729517 26525 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:03.729701 26510 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:09:03.729779 26510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "2af81dc46d90481cb4c9634019878ea4"
format_stamp: "Formatted at 2025-10-26 15:09:03 on dist-test-slave-8pgz"
server_key: "c5524bad71b0deb2ddf13bcd80932223"
server_key_iv: "f061b1d18af56d4d110602ccae5bec2a"
server_key_version: "encryptionkey@0"
I20251026 15:09:03.729842 26510 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:09:03.741246 26510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:09:03.743577 26510 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:09:03.743722 26510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:09:03.743937 26510 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:09:03.744297 26510 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:09:03.744334 26510 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:03.744359 26510 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:09:03.744371 26510 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:09:03.749998 26510 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:46777
I20251026 15:09:03.750054 26638 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:46777 every 8 connection(s)
I20251026 15:09:03.750348 26510 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:09:03.754269 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 26510
I20251026 15:09:03.754372 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:09:03.754594 15185 external_mini_cluster.cc:1468] Setting key ef7861875b9af498f7db11e7aab90809
Oct 26 15:09:03 dist-test-slave-8pgz krb5kdc[26126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:09:03.759197 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.751084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:59139 (local address 127.14.212.126:40829)
1026 15:09:03.751146 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:03.751149 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:03.751340 (+ 191us) server_negotiation.cc:408] Connection header received
1026 15:09:03.751856 (+ 516us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:03.751857 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:03.751869 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:03.751926 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:03.752720 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.753205 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.753799 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.753966 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.756624 (+ 2658us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:09:03.756627 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:09:03.756628 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:09:03.756634 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:09:03.758038 (+ 1404us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:09:03.758461 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:09:03.758461 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:09:03.758462 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:09:03.758481 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:09:03.758631 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:09:03.758632 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:09:03.758632 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:09:03.758720 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:09:03.758793 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:03.759052 (+ 259us) server_negotiation.cc:300] Negotiation successful
1026 15:09:03.759129 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251026 15:09:03.759333 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.751021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:40829 (local address 127.14.212.67:59139)
1026 15:09:03.751235 (+ 214us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:03.751261 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:03.751765 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:03.752005 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:03.752008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:03.752351 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:09:03.752627 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.752630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.753284 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.753285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:09:03.753720 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.753722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.753865 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.755038 (+ 1173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:09:03.755060 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:09:03.756541 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:09:03.758132 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:09:03.758138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:09:03.758141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:09:03.758345 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:09:03.758532 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:09:03.758532 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:09:03.758533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:09:03.758587 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:09:03.758816 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:09:03.758821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:09:03.758939 (+ 118us) client_negotiation.cc:769] Sending connection context
1026 15:09:03.759025 (+ 86us) client_negotiation.cc:240] Negotiation successful
1026 15:09:03.759128 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20251026 15:09:03.759819 26639 heartbeater.cc:344] Connected to a master server at 127.14.212.126:40829
I20251026 15:09:03.759909 26639 heartbeater.cc:461] Registering TS with master...
I20251026 15:09:03.760095 26639 heartbeater.cc:507] Master 127.14.212.126:40829 requested a full tablet report, sending...
I20251026 15:09:03.760564 26174 ts_manager.cc:194] Registered new tserver with Master: 2af81dc46d90481cb4c9634019878ea4 (127.14.212.67:46777)
I20251026 15:09:03.760968 26174 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:59139
I20251026 15:09:03.766696 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:09:03.773775 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.768066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.14.212.126:40829)
1026 15:09:03.768137 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:03.768139 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:03.768197 (+ 58us) server_negotiation.cc:408] Connection header received
1026 15:09:03.768263 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:03.768264 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:03.768275 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:03.768332 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:09:03.769025 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.769517 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.770065 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.770263 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.771293 (+ 1030us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:09:03.771297 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:09:03.771297 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:09:03.771303 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:09:03.772568 (+ 1265us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:09:03.773037 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:09:03.773038 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:09:03.773039 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:09:03.773068 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:09:03.773209 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:09:03.773210 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:09:03.773211 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:09:03.773355 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:09:03.773465 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:03.773573 (+ 108us) server_negotiation.cc:300] Negotiation successful
1026 15:09:03.773664 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251026 15:09:03.775357 26174 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:09:03.776634 26174 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:09:03.781656 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.778898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:36425 (local address 127.0.0.1:52718)
1026 15:09:03.779224 (+ 326us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:03.779235 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:03.779332 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:03.779548 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:03.779551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:03.779553 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:09:03.779832 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.779834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.780703 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.780704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:09:03.781371 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.781372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.781442 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.781480 (+ 38us) client_negotiation.cc:769] Sending connection context
1026 15:09:03.781531 (+ 51us) client_negotiation.cc:240] Negotiation successful
1026 15:09:03.781568 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":82,"threads_started":1}
I20251026 15:09:03.782310 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.779005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.14.212.66:36425)
1026 15:09:03.779189 (+ 184us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:03.779202 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:03.779302 (+ 100us) server_negotiation.cc:408] Connection header received
1026 15:09:03.779416 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:03.779419 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:03.779438 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:03.779511 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:09:03.779920 (+ 409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.780618 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.781468 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.782149 (+ 681us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.782193 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:03.782219 (+ 26us) server_negotiation.cc:300] Negotiation successful
1026 15:09:03.782253 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":88,"threads_started":1}
I20251026 15:09:03.782721 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.778897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:42937 (local address 127.0.0.1:36070)
1026 15:09:03.779054 (+ 157us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:03.779083 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:03.779160 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:03.779729 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:03.779730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:03.779732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:09:03.779902 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.779903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.781214 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.781215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:09:03.782507 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.782509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.782569 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.782572 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:09:03.782607 (+ 35us) client_negotiation.cc:240] Negotiation successful
1026 15:09:03.782644 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":59,"threads_started":1}
I20251026 15:09:03.782871 26437 tablet_service.cc:1505] Processing CreateTablet for tablet 0e83e717c1a44907873fd57f49fea501 (DEFAULT_TABLE table=test-table [id=42ad652c812b4e08bc5a14209405d340]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:09:03.783156 26437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e83e717c1a44907873fd57f49fea501. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:09:03.783519 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.779319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.14.212.67:46777)
1026 15:09:03.779471 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:03.779479 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:03.779488 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:09:03.779532 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:03.779534 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:03.779547 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:03.779608 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:09:03.780002 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.781136 (+ 1134us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.782760 (+ 1624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.783315 (+ 555us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.783366 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:03.783397 (+ 31us) server_negotiation.cc:300] Negotiation successful
1026 15:09:03.783439 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20251026 15:09:03.783705 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.778979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36070 (local address 127.14.212.65:42937)
1026 15:09:03.779478 (+ 499us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:03.779485 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:03.779497 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:09:03.779537 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:03.779539 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:03.779556 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:03.779648 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:09:03.779984 (+ 336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.781126 (+ 1142us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.782959 (+ 1833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.783512 (+ 553us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.783559 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:03.783590 (+ 31us) server_negotiation.cc:300] Negotiation successful
1026 15:09:03.783641 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":77,"threads_started":1}
I20251026 15:09:03.782722 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.779095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:46777 (local address 127.0.0.1:35580)
1026 15:09:03.779298 (+ 203us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:03.779308 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:03.779360 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:03.779690 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:03.779691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:03.779693 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:09:03.779912 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.779914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.781607 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.781608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:09:03.782485 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.782497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.782570 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.782572 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:09:03.782607 (+ 35us) client_negotiation.cc:240] Negotiation successful
1026 15:09:03.782644 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":58,"threads_started":1}
I20251026 15:09:03.784339 26301 tablet_service.cc:1505] Processing CreateTablet for tablet 0e83e717c1a44907873fd57f49fea501 (DEFAULT_TABLE table=test-table [id=42ad652c812b4e08bc5a14209405d340]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:09:03.784523 26573 tablet_service.cc:1505] Processing CreateTablet for tablet 0e83e717c1a44907873fd57f49fea501 (DEFAULT_TABLE table=test-table [id=42ad652c812b4e08bc5a14209405d340]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:09:03.784655 26301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e83e717c1a44907873fd57f49fea501. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:09:03.784781 26573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e83e717c1a44907873fd57f49fea501. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:09:03.786072 26656 tablet_bootstrap.cc:492] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd: Bootstrap starting.
I20251026 15:09:03.786715 26656 tablet_bootstrap.cc:654] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd: Neither blocks nor log segments found. Creating new log.
I20251026 15:09:03.787048 26656 log.cc:826] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd: Log is configured to *not* fsync() on all Append() calls
I20251026 15:09:03.787195 26657 tablet_bootstrap.cc:492] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4: Bootstrap starting.
I20251026 15:09:03.787485 26658 tablet_bootstrap.cc:492] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1: Bootstrap starting.
I20251026 15:09:03.787842 26657 tablet_bootstrap.cc:654] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4: Neither blocks nor log segments found. Creating new log.
I20251026 15:09:03.787976 26656 tablet_bootstrap.cc:492] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd: No bootstrap required, opened a new log
I20251026 15:09:03.788013 26658 tablet_bootstrap.cc:654] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1: Neither blocks nor log segments found. Creating new log.
I20251026 15:09:03.788053 26656 ts_tablet_manager.cc:1403] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 15:09:03.788187 26657 log.cc:826] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4: Log is configured to *not* fsync() on all Append() calls
I20251026 15:09:03.788273 26658 log.cc:826] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1: Log is configured to *not* fsync() on all Append() calls
I20251026 15:09:03.788827 26657 tablet_bootstrap.cc:492] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4: No bootstrap required, opened a new log
I20251026 15:09:03.788869 26657 ts_tablet_manager.cc:1403] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:09:03.788918 26658 tablet_bootstrap.cc:492] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1: No bootstrap required, opened a new log
I20251026 15:09:03.788971 26658 ts_tablet_manager.cc:1403] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:09:03.789914 26657 raft_consensus.cc:359] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58273c39e8674f2b91747ec44b07badd" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36425 } } peers { permanent_uuid: "f1b041cb538f437dac8cfa44cb717dd1" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 42937 } } peers { permanent_uuid: "2af81dc46d90481cb4c9634019878ea4" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46777 } }
I20251026 15:09:03.789930 26656 raft_consensus.cc:359] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58273c39e8674f2b91747ec44b07badd" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36425 } } peers { permanent_uuid: "f1b041cb538f437dac8cfa44cb717dd1" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 42937 } } peers { permanent_uuid: "2af81dc46d90481cb4c9634019878ea4" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46777 } }
I20251026 15:09:03.790056 26656 raft_consensus.cc:385] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:09:03.790057 26657 raft_consensus.cc:385] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:09:03.790086 26656 raft_consensus.cc:740] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58273c39e8674f2b91747ec44b07badd, State: Initialized, Role: FOLLOWER
I20251026 15:09:03.790091 26657 raft_consensus.cc:740] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af81dc46d90481cb4c9634019878ea4, State: Initialized, Role: FOLLOWER
I20251026 15:09:03.790172 26656 consensus_queue.cc:260] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58273c39e8674f2b91747ec44b07badd" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36425 } } peers { permanent_uuid: "f1b041cb538f437dac8cfa44cb717dd1" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 42937 } } peers { permanent_uuid: "2af81dc46d90481cb4c9634019878ea4" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46777 } }
I20251026 15:09:03.790181 26657 consensus_queue.cc:260] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58273c39e8674f2b91747ec44b07badd" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36425 } } peers { permanent_uuid: "f1b041cb538f437dac8cfa44cb717dd1" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 42937 } } peers { permanent_uuid: "2af81dc46d90481cb4c9634019878ea4" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46777 } }
I20251026 15:09:03.790458 26656 ts_tablet_manager.cc:1434] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:09:03.790539 26639 heartbeater.cc:499] Master 127.14.212.126:40829 was elected leader, sending a full tablet report...
I20251026 15:09:03.790609 26657 ts_tablet_manager.cc:1434] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:09:03.790642 26503 heartbeater.cc:499] Master 127.14.212.126:40829 was elected leader, sending a full tablet report...
I20251026 15:09:03.791056 26658 raft_consensus.cc:359] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58273c39e8674f2b91747ec44b07badd" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36425 } } peers { permanent_uuid: "f1b041cb538f437dac8cfa44cb717dd1" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 42937 } } peers { permanent_uuid: "2af81dc46d90481cb4c9634019878ea4" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46777 } }
I20251026 15:09:03.791190 26658 raft_consensus.cc:385] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:09:03.791278 26658 raft_consensus.cc:740] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b041cb538f437dac8cfa44cb717dd1, State: Initialized, Role: FOLLOWER
I20251026 15:09:03.791414 26658 consensus_queue.cc:260] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58273c39e8674f2b91747ec44b07badd" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36425 } } peers { permanent_uuid: "f1b041cb538f437dac8cfa44cb717dd1" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 42937 } } peers { permanent_uuid: "2af81dc46d90481cb4c9634019878ea4" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46777 } }
I20251026 15:09:03.791687 26658 ts_tablet_manager.cc:1434] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251026 15:09:03.791775 26367 heartbeater.cc:499] Master 127.14.212.126:40829 was elected leader, sending a full tablet report...
W20251026 15:09:03.826300 26504 tablet.cc:2378] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:09:03.844689 26662 raft_consensus.cc:493] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:09:03.844818 26662 raft_consensus.cc:515] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58273c39e8674f2b91747ec44b07badd" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36425 } } peers { permanent_uuid: "f1b041cb538f437dac8cfa44cb717dd1" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 42937 } } peers { permanent_uuid: "2af81dc46d90481cb4c9634019878ea4" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46777 } }
I20251026 15:09:03.845139 26662 leader_election.cc:290] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58273c39e8674f2b91747ec44b07badd (127.14.212.66:36425), f1b041cb538f437dac8cfa44cb717dd1 (127.14.212.65:42937)
I20251026 15:09:03.847353 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.845351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:42937 (local address 127.14.212.67:58327)
1026 15:09:03.845439 (+ 88us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:03.845452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:03.845509 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:03.845680 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:03.845680 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:03.845682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:09:03.845866 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.845868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.846558 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.846559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:09:03.847127 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.847128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.847200 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.847204 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:09:03.847238 (+ 34us) client_negotiation.cc:240] Negotiation successful
1026 15:09:03.847275 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20251026 15:09:03.847575 26665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.845393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:36425 (local address 127.14.212.67:51681)
1026 15:09:03.845602 (+ 209us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:03.845612 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:09:03.845684 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:09:03.845826 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:09:03.845827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:09:03.845828 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:09:03.846048 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.846049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.846745 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.846746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:09:03.847334 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:09:03.847336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.847409 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.847413 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:09:03.847446 (+ 33us) client_negotiation.cc:240] Negotiation successful
1026 15:09:03.847490 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":140,"threads_started":1}
I20251026 15:09:03.847834 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.845413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:58327 (local address 127.14.212.65:42937)
1026 15:09:03.845457 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:03.845460 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:03.845491 (+ 31us) server_negotiation.cc:408] Connection header received
1026 15:09:03.845601 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:03.845602 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:03.845611 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:03.845663 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:09:03.845937 (+ 274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.846452 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.847217 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.847714 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.847724 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:03.847747 (+ 23us) server_negotiation.cc:300] Negotiation successful
1026 15:09:03.847782 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251026 15:09:03.848033 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.845501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:51681 (local address 127.14.212.66:36425)
1026 15:09:03.845548 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:03.845550 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:03.845699 (+ 149us) server_negotiation.cc:408] Connection header received
1026 15:09:03.845749 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:03.845750 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:03.845759 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:03.845810 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:09:03.846131 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.846663 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.847421 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.847909 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.847918 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:03.847945 (+ 27us) server_negotiation.cc:300] Negotiation successful
1026 15:09:03.847983 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251026 15:09:03.848145 26321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e83e717c1a44907873fd57f49fea501" candidate_uuid: "2af81dc46d90481cb4c9634019878ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1b041cb538f437dac8cfa44cb717dd1" is_pre_election: true
I20251026 15:09:03.848309 26321 raft_consensus.cc:2468] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2af81dc46d90481cb4c9634019878ea4 in term 0.
I20251026 15:09:03.848348 26457 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e83e717c1a44907873fd57f49fea501" candidate_uuid: "2af81dc46d90481cb4c9634019878ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58273c39e8674f2b91747ec44b07badd" is_pre_election: true
I20251026 15:09:03.848505 26457 raft_consensus.cc:2468] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2af81dc46d90481cb4c9634019878ea4 in term 0.
I20251026 15:09:03.848544 26527 leader_election.cc:304] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2af81dc46d90481cb4c9634019878ea4, f1b041cb538f437dac8cfa44cb717dd1; no voters:
I20251026 15:09:03.848667 26662 raft_consensus.cc:2804] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:09:03.848728 26662 raft_consensus.cc:493] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:09:03.848758 26662 raft_consensus.cc:3060] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:09:03.849436 26662 raft_consensus.cc:515] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58273c39e8674f2b91747ec44b07badd" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36425 } } peers { permanent_uuid: "f1b041cb538f437dac8cfa44cb717dd1" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 42937 } } peers { permanent_uuid: "2af81dc46d90481cb4c9634019878ea4" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46777 } }
I20251026 15:09:03.849578 26662 leader_election.cc:290] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [CANDIDATE]: Term 1 election: Requested vote from peers 58273c39e8674f2b91747ec44b07badd (127.14.212.66:36425), f1b041cb538f437dac8cfa44cb717dd1 (127.14.212.65:42937)
I20251026 15:09:03.849756 26457 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e83e717c1a44907873fd57f49fea501" candidate_uuid: "2af81dc46d90481cb4c9634019878ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58273c39e8674f2b91747ec44b07badd"
I20251026 15:09:03.849771 26321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e83e717c1a44907873fd57f49fea501" candidate_uuid: "2af81dc46d90481cb4c9634019878ea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1b041cb538f437dac8cfa44cb717dd1"
I20251026 15:09:03.849841 26457 raft_consensus.cc:3060] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:09:03.849841 26321 raft_consensus.cc:3060] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:09:03.850775 26321 raft_consensus.cc:2468] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2af81dc46d90481cb4c9634019878ea4 in term 1.
I20251026 15:09:03.850775 26457 raft_consensus.cc:2468] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2af81dc46d90481cb4c9634019878ea4 in term 1.
I20251026 15:09:03.850992 26528 leader_election.cc:304] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2af81dc46d90481cb4c9634019878ea4, 58273c39e8674f2b91747ec44b07badd; no voters:
I20251026 15:09:03.851110 26662 raft_consensus.cc:2804] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:09:03.851287 26662 raft_consensus.cc:697] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [term 1 LEADER]: Becoming Leader. State: Replica: 2af81dc46d90481cb4c9634019878ea4, State: Running, Role: LEADER
I20251026 15:09:03.851405 26662 consensus_queue.cc:237] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58273c39e8674f2b91747ec44b07badd" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36425 } } peers { permanent_uuid: "f1b041cb538f437dac8cfa44cb717dd1" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 42937 } } peers { permanent_uuid: "2af81dc46d90481cb4c9634019878ea4" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46777 } }
I20251026 15:09:03.852216 26173 catalog_manager.cc:5649] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2af81dc46d90481cb4c9634019878ea4 (127.14.212.67). New cstate: current_term: 1 leader_uuid: "2af81dc46d90481cb4c9634019878ea4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58273c39e8674f2b91747ec44b07badd" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1b041cb538f437dac8cfa44cb717dd1" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2af81dc46d90481cb4c9634019878ea4" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46777 } health_report { overall_health: HEALTHY } } }
I20251026 15:09:03.860864 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.858800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.14.212.67:46777)
1026 15:09:03.858846 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1026 15:09:03.858848 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:09:03.858858 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:09:03.858895 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:09:03.858896 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:09:03.858911 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:09:03.858967 (+ 56us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:09:03.859242 (+ 275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.859730 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:09:03.860325 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:09:03.860468 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:09:03.860505 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:09:03.860604 (+ 99us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:09:03.860681 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1026 15:09:03.860752 (+ 71us) server_negotiation.cc:300] Negotiation successful
1026 15:09:03.860779 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251026 15:09:03.862635 26457 raft_consensus.cc:1275] T 0e83e717c1a44907873fd57f49fea501 P 58273c39e8674f2b91747ec44b07badd [term 1 FOLLOWER]: Refusing update from remote peer 2af81dc46d90481cb4c9634019878ea4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:09:03.862634 26321 raft_consensus.cc:1275] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1 [term 1 FOLLOWER]: Refusing update from remote peer 2af81dc46d90481cb4c9634019878ea4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:09:03.862957 26666 consensus_queue.cc:1048] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1b041cb538f437dac8cfa44cb717dd1" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:09:03.863082 26662 consensus_queue.cc:1048] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58273c39e8674f2b91747ec44b07badd" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:09:03.865485 26671 mvcc.cc:204] Tried to move back new op lower bound from 7215068544458694656 to 7215068544416047104. Current Snapshot: MvccSnapshot[applied={T|T < 7215068544458694656}]
I20251026 15:09:03.865929 26670 mvcc.cc:204] Tried to move back new op lower bound from 7215068544458694656 to 7215068544416047104. Current Snapshot: MvccSnapshot[applied={T|T < 7215068544458694656}]
I20251026 15:09:03.867653 26174 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57502:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:09:03.867743 26174 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57502:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:09:03.868553 26174 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 97ef8bbb72374146a3012ce787c01f38: Sending DeleteTablet for 3 replicas of tablet 0e83e717c1a44907873fd57f49fea501
I20251026 15:09:03.868891 26437 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e83e717c1a44907873fd57f49fea501 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:09:03 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:52718
I20251026 15:09:03.868897 26573 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e83e717c1a44907873fd57f49fea501 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:09:03 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:35580
I20251026 15:09:03.869053 26301 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e83e717c1a44907873fd57f49fea501 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:09:03 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:36070
I20251026 15:09:03.869277 26680 tablet_replica.cc:333] T 0e83e717c1a44907873fd57f49fea501 P f1b041cb538f437dac8cfa44cb717dd1: stopping tablet replica
I20251026 15:09:03.869418 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 26238
I20251026 15:09:03.869433 26679 tablet_replica.cc:333] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4: stopping tablet replica
I20251026 15:09:03.870291 26679 raft_consensus.cc:2243] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:09:03.870471 26679 raft_consensus.cc:2272] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:09:03.870949 26679 ts_tablet_manager.cc:1916] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:09:03.872617 26679 ts_tablet_manager.cc:1929] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:09:03.872694 26679 log.cc:1199] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/0e83e717c1a44907873fd57f49fea501
I20251026 15:09:03.872964 26679 ts_tablet_manager.cc:1950] T 0e83e717c1a44907873fd57f49fea501 P 2af81dc46d90481cb4c9634019878ea4: Deleting consensus metadata
I20251026 15:09:03.873404 26160 catalog_manager.cc:4985] TS 2af81dc46d90481cb4c9634019878ea4 (127.14.212.67:46777): tablet 0e83e717c1a44907873fd57f49fea501 (table test-table [id=42ad652c812b4e08bc5a14209405d340]) successfully deleted
W20251026 15:09:03.873908 26159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:42937 (error 108)
I20251026 15:09:03.874430 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.874278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:42937 (local address 127.0.0.1:36080)
1026 15:09:03.874330 (+ 52us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:03.874365 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20251026 15:09:03.874610 26159 catalog_manager.cc:4712] TS f1b041cb538f437dac8cfa44cb717dd1 (127.14.212.65:42937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e83e717c1a44907873fd57f49fea501: Network error: Client connection negotiation failed: client connection to 127.14.212.65:42937: connect: Connection refused (error 111)
I20251026 15:09:03.875047 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 26374
I20251026 15:09:03.883016 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:09:03.882863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:36425 (local address 127.0.0.1:52732)
1026 15:09:03.882938 (+ 75us) negotiation.cc:107] Waiting for socket to connect
1026 15:09:03.882967 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.66:36425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20251026 15:09:03.883014 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 26510
I20251026 15:09:03.887952 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 26142
2025-10-26T15:09:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3013 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:09:03 dist-test-slave-8pgz krb5kdc[26684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:09:03 dist-test-slave-8pgz krb5kdc[26684](info): set up 2 sockets
Oct 26 15:09:03 dist-test-slave-8pgz krb5kdc[26684](info): commencing operation
krb5kdc: starting...
W20251026 15:09:05.876497 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.968s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:09:05 dist-test-slave-8pgz krb5kdc[26684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:09:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:09:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.212.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.212.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/krb5kdc/rangeradmin_127.14.212.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/krb5kdc/rangeradmin_127.14.212.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.212.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.212.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/krb5kdc/rangerlookup_127.14.212.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/krb5kdc/rangerlookup_127.14.212.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.212.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/krb5kdc/HTTP_127.14.212.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/krb5kdc/HTTP_127.14.212.84@KRBTEST.COM.keytab.
I20251026 15:09:06.037135 15185 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskXR9jZt/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/minicluster-data/postgres -l logfile start
2025-10-26 15:09:08.709 UTC [26715] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-10-26 15:09:08.709 UTC [26715] LOG: listening on IPv4 address "127.14.212.84", port 60419
2025-10-26 15:09:08.714 UTC [26715] LOG: listening on Unix socket "/tmp/.s.PGSQL.60419"
2025-10-26 15:09:08.720 UTC [26720] LOG: database system was shut down at 2025-10-26 15:09:07 UTC
2025-10-26 15:09:08.725 UTC [26715] LOG: database system is ready to accept connections
I20251026 15:09:10.624558 15185 mini_postgres.cc:96] Postgres bound to 60419
2025-10-26 15:09:10.630 UTC [26726] FATAL: database "slave" does not exist
127.14.212.84:60419 - accepting connections
I20251026 15:09:10.631373 15185 mini_ranger.cc:161] Starting Ranger...
I20251026 15:09:10.649731 15185 mini_ranger.cc:84] Created miniranger Postgres user
I20251026 15:09:10.724870 15185 mini_ranger.cc:87] Created ranger Postgres database
I20251026 15:09:10.724974 15185 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/minicluster-data/ranger-admin
2025-10-26 15:09:11,025 [I] DB FLAVOR :POSTGRES
2025-10-26 15:09:11,026 [I] --------- Verifying Ranger DB connection ---------
2025-10-26 15:09:11,026 [I] Checking connection..
2025-10-26 15:09:11,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-10-26 15:09:11,350 [I] Checking connection passed.
2025-10-26 15:09:11,351 [I] --------- Verifying version history table ---------
2025-10-26 15:09:11,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-26 15:09:11,687 [I] Table x_db_version_h does not exist in database ranger
2025-10-26 15:09:11,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-26 15:09:12,016 [I] Table x_db_version_h does not exist in database ranger
2025-10-26 15:09:12,016 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-10-26 15:09:12,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-10-26 15:09:12.337 UTC [26793] WARNING: there is no transaction in progress
2025-10-26 15:09:12,344 [I] create_dbversion_catalog.sql file imported successfully
2025-10-26 15:09:12,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-26 15:09:12,645 [I] Table x_db_version_h already exists in database 'ranger'
2025-10-26 15:09:12,645 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-26 15:09:12,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-10-26 15:09:12,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-10-26 15:09:13,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8pgz.c.gcp-upstream.internal','N') ;"
2025-10-26 15:09:13,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-26 15:09:13,921 [I] Table x_portal_user does not exist in database ranger
2025-10-26 15:09:13,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-26 15:09:14,267 [I] Table x_policy_ref_group does not exist in database ranger
2025-10-26 15:09:14,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-26 15:09:14,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-26 15:09:14,902 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-10-26 15:09:14,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-10-26 15:09:15.776 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.787 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.799 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.810 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.823 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.835 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.884 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.890 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.900 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.910 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.919 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.931 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.939 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.945 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.949 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:15.952 UTC [26910] WARNING: there is no transaction in progress
2025-10-26 15:09:16,375 [I] ranger_core_db_postgres.sql file imported successfully
2025-10-26 15:09:16,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-10-26 15:09:16,712 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-10-26 15:09:16,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-26 15:09:17,018 [I] Table x_portal_user already exists in database 'ranger'
2025-10-26 15:09:17,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-26 15:09:17,356 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-10-26 15:09:17,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-26 15:09:17,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-26 15:09:18,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8pgz.c.gcp-upstream.internal';"
2025-10-26 15:09:18,325 [I] CORE_DB_SCHEMA import status has been updated
2025-10-26 15:09:18,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:60419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-10-26 15:09:18,645 [I] DB_PATCHES have already been applied
I20251026 15:09:18.651630 15185 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/minicluster-data/ranger-admin:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskXR9jZt/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251026 15:09:18.651705 15185 mini_ranger.cc:193] Using host: 127.14.212.84
I20251026 15:09:18.654557 15185 mini_ranger.cc:227] Ranger admin URL: http://127.14.212.84:46783
Oct 26, 2025 3:09:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 26, 2025 3:09:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 26, 2025 3:09:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/minicluster-data/ranger-admin/ews/webapp
Oct 26, 2025 3:09:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Oct 26, 2025 3:09:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/minicluster-data/ranger-admin/ews/webapp] .....
Oct 26, 2025 3:09:19 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Oct 26, 2025 3:09:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0/minicluster-data/ranger-admin/ews/webapp].
Oct 26, 2025 3:09:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Oct 26, 2025 3:09:19 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-46783"]
Oct 26, 2025 3:09:19 PM org.apache.coyote.AbstractProtocol init
SEVERE: Failed to initialize end point associated with ProtocolHandler ["http-bio-46783"]
java.net.BindException: Address already in use (Bind failed) <null>:46783
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:414)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:750)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:467)
at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:119)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1030)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:560)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:826)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:173)
at org.apache.catalina.startup.Tomcat.start(Tomcat.java:469)
at org.apache.ranger.server.tomcat.EmbeddedServer.startServer(EmbeddedServer.java:322)
at org.apache.ranger.server.tomcat.EmbeddedServer.start(EmbeddedServer.java:290)
at org.apache.ranger.server.tomcat.EmbeddedServer.main(EmbeddedServer.java:84)
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:390)
at java.net.ServerSocket.<init>(ServerSocket.java:252)
at java.net.ServerSocket.<init>(ServerSocket.java:196)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocketFactory.java:49)
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:401)
... 14 more
Oct 26, 2025 3:09:19 PM org.apache.catalina.core.StandardService initInternal
SEVERE: Failed to initialize connector [Connector[HTTP/1.1-46783]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1032)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:560)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:826)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:173)
at org.apache.catalina.startup.Tomcat.start(Tomcat.java:469)
at org.apache.ranger.server.tomcat.EmbeddedServer.startServer(EmbeddedServer.java:322)
at org.apache.ranger.server.tomcat.EmbeddedServer.start(EmbeddedServer.java:290)
at org.apache.ranger.server.tomcat.EmbeddedServer.main(EmbeddedServer.java:84)
Caused by: java.net.BindException: Address already in use (Bind failed) <null>:46783
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:414)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:750)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:467)
at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:119)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1030)
... 10 more
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:390)
at java.net.ServerSocket.<init>(ServerSocket.java:252)
at java.net.ServerSocket.<init>(ServerSocket.java:196)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocketFactory.java:49)
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:401)
... 14 more
Oct 26, 2025 3:09:19 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 26, 2025 3:09:19 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 26, 2025 3:09:19 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Oct 26, 2025 3:09:19 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Oct 26, 2025 3:09:19 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 26, 2025 3:09:20 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Oct 26, 2025 3:09:20 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 26, 2025 3:09:20 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Oct 26, 2025 3:09:20 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Oct 26, 2025 3:09:45 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Oct 26, 2025 3:09:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
Oct 26, 2025 3:09:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Oct 26, 2025 3:09:45 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Oct 26, 2025 3:09:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Oct 26, 2025 3:09:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Oct 26, 2025 3:09:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Oct 26, 2025 3:09:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Oct 26, 2025 3:09:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Oct 26, 2025 3:09:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Oct 26, 2025 3:09:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Oct 26, 2025 3:09:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Oct 26, 2025 3:09:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Oct 26, 2025 3:09:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Oct 26, 2025 3:09:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Oct 26, 2025 3:09:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Oct 26, 2025 3:09:45 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Oct 26, 2025 3:09:46 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20251026 15:11:19.343449 15185 mini_ranger.cc:160] Time spent starting Ranger: real 128.712s user 0.004s sys 0.088s
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/security-itest.cc:1052: Failure
Failed
Bad status: Timed out: Failed to start the Ranger service: process with PID 27004 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p27004
f310
n127.0.0.1:57886->127.14.212.84:60419
f311
n127.0.0.1:57878->127.14.212.84:60419
f312
n127.0.0.1:57902->127.14.212.84:60419
f313
n127.0.0.1:57916->127.14.212.84:60419
f314
n127.0.0.1:57926->127.14.212.84:60419'
I20251026 15:11:19.344425 15185 mini_ranger.cc:66] Stopping Ranger...
I20251026 15:11:19.376150 15185 mini_ranger.cc:68] Stopped Ranger
2025-10-26 15:11:19.376 UTC [26715] LOG: received smart shutdown request
2025-10-26 15:11:19.380 UTC [26715] LOG: background worker "logical replication launcher" (PID 26723) exited with exit code 1
2025-10-26 15:11:19.380 UTC [26718] LOG: shutting down
2025-10-26 15:11:19.382 UTC [26718] LOG: checkpoint starting: shutdown immediate
2025-10-26 15:11:19.864 UTC [26718] LOG: checkpoint complete: wrote 1468 buffers (9.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.026 s, sync=0.451 s, total=0.484 s; sync files=815, longest=0.005 s, average=0.001 s; distance=8396 kB, estimate=8396 kB; lsn=0/1CADC10, redo lsn=0/1CADC10
2025-10-26 15:11:19.871 UTC [26715] LOG: database system is shut down
2025-10-26T15:11:19Z chronyd exiting
I20251026 15:11:19.884569 15185 test_util.cc:183] -----------------------------------------------
I20251026 15:11:19.884647 15185 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761491252308956-15185-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (135976 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:11:19 dist-test-slave-8pgz krb5kdc[27148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:11:19 dist-test-slave-8pgz krb5kdc[27148](info): set up 2 sockets
Oct 26 15:11:19 dist-test-slave-8pgz krb5kdc[27148](info): commencing operation
krb5kdc: starting...
W20251026 15:11:21.863343 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.978s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:11:21 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:11:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:11:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.212.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.212.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/rangeradmin_127.14.212.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/rangeradmin_127.14.212.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.212.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.212.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/rangerlookup_127.14.212.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/rangerlookup_127.14.212.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.212.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/HTTP_127.14.212.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/HTTP_127.14.212.84@KRBTEST.COM.keytab.
I20251026 15:11:22.029168 15185 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskXR9jZt/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/postgres -l logfile start
2025-10-26 15:11:24.761 UTC [27179] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-10-26 15:11:24.761 UTC [27179] LOG: listening on IPv4 address "127.14.212.84", port 49023
2025-10-26 15:11:24.765 UTC [27179] LOG: listening on Unix socket "/tmp/.s.PGSQL.49023"
2025-10-26 15:11:24.771 UTC [27184] LOG: database system was shut down at 2025-10-26 15:11:23 UTC
2025-10-26 15:11:24.776 UTC [27179] LOG: database system is ready to accept connections
I20251026 15:11:26.679322 15185 mini_postgres.cc:96] Postgres bound to 49023
2025-10-26 15:11:26.685 UTC [27190] FATAL: database "slave" does not exist
127.14.212.84:49023 - accepting connections
I20251026 15:11:26.685631 15185 mini_ranger.cc:161] Starting Ranger...
I20251026 15:11:26.703567 15185 mini_ranger.cc:84] Created miniranger Postgres user
I20251026 15:11:26.774511 15185 mini_ranger.cc:87] Created ranger Postgres database
I20251026 15:11:26.774597 15185 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-admin
2025-10-26 15:11:27,073 [I] DB FLAVOR :POSTGRES
2025-10-26 15:11:27,073 [I] --------- Verifying Ranger DB connection ---------
2025-10-26 15:11:27,073 [I] Checking connection..
2025-10-26 15:11:27,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-10-26 15:11:27,396 [I] Checking connection passed.
2025-10-26 15:11:27,396 [I] --------- Verifying version history table ---------
2025-10-26 15:11:27,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-26 15:11:27,722 [I] Table x_db_version_h does not exist in database ranger
2025-10-26 15:11:27,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-26 15:11:28,067 [I] Table x_db_version_h does not exist in database ranger
2025-10-26 15:11:28,068 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-10-26 15:11:28,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-10-26 15:11:28.390 UTC [27257] WARNING: there is no transaction in progress
2025-10-26 15:11:28,397 [I] create_dbversion_catalog.sql file imported successfully
2025-10-26 15:11:28,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-26 15:11:28,713 [I] Table x_db_version_h already exists in database 'ranger'
2025-10-26 15:11:28,714 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-26 15:11:28,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-10-26 15:11:29,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-10-26 15:11:29,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8pgz.c.gcp-upstream.internal','N') ;"
2025-10-26 15:11:29,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-26 15:11:30,067 [I] Table x_portal_user does not exist in database ranger
2025-10-26 15:11:30,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-26 15:11:30,414 [I] Table x_policy_ref_group does not exist in database ranger
2025-10-26 15:11:30,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-26 15:11:30,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-26 15:11:31,084 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-10-26 15:11:31,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-10-26 15:11:31.972 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:31.987 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.003 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.018 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.034 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.046 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.097 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.103 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.113 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.124 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.133 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.144 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.153 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.158 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.162 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32.166 UTC [27374] WARNING: there is no transaction in progress
2025-10-26 15:11:32,582 [I] ranger_core_db_postgres.sql file imported successfully
2025-10-26 15:11:32,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-10-26 15:11:32,902 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-10-26 15:11:32,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-26 15:11:33,242 [I] Table x_portal_user already exists in database 'ranger'
2025-10-26 15:11:33,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-26 15:11:33,549 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-10-26 15:11:33,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-26 15:11:33,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-26 15:11:34,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8pgz.c.gcp-upstream.internal';"
2025-10-26 15:11:34,550 [I] CORE_DB_SCHEMA import status has been updated
2025-10-26 15:11:34,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-10-26 15:11:34,877 [I] DB_PATCHES have already been applied
I20251026 15:11:34.883562 15185 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-admin:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskXR9jZt/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251026 15:11:34.883637 15185 mini_ranger.cc:193] Using host: 127.14.212.84
I20251026 15:11:34.886222 15185 mini_ranger.cc:227] Ranger admin URL: http://127.14.212.84:40725
Oct 26, 2025 3:11:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 26, 2025 3:11:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 26, 2025 3:11:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-admin/ews/webapp
Oct 26, 2025 3:11:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Oct 26, 2025 3:11:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-admin/ews/webapp] .....
Oct 26, 2025 3:11:35 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Oct 26, 2025 3:11:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-admin/ews/webapp].
Oct 26, 2025 3:11:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Oct 26, 2025 3:11:35 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-40725"]
Oct 26, 2025 3:11:35 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 26, 2025 3:11:35 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 26, 2025 3:11:36 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Oct 26, 2025 3:11:36 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Oct 26, 2025 3:11:36 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 26, 2025 3:11:36 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Oct 26, 2025 3:11:36 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 26, 2025 3:11:37 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Oct 26, 2025 3:11:37 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
I20251026 15:11:37.132089 15185 mini_ranger.cc:160] Time spent starting Ranger: real 10.446s user 0.000s sys 0.009s
Oct 26, 2025 3:11:57 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Oct 26, 2025 3:11:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Oct 26, 2025 3:11:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Oct 26, 2025 3:11:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Oct 26, 2025 3:11:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Oct 26, 2025 3:11:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Oct 26, 2025 3:11:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Oct 26, 2025 3:11:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Oct 26, 2025 3:11:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Oct 26, 2025 3:11:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Oct 26, 2025 3:11:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Oct 26, 2025 3:11:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Oct 26, 2025 3:11:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Oct 26, 2025 3:11:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Oct 26, 2025 3:11:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Oct 26, 2025 3:11:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Oct 26, 2025 3:11:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Oct 26, 2025 3:11:58 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Oct 26, 2025 3:11:58 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-40725"]
I20251026 15:11:59.123842 15185 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.212.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.212.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/rangerkms_127.14.212.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.212.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/rangerkms_127.14.212.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.212.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.212.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/HTTP_127.14.212.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.212.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/HTTP_127.14.212.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Oct 26 15:11:59 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491519, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251026 15:11:59.205994 15185 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251026 15:11:59.216553 15185 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251026 15:11:59.324091 15185 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251026 15:11:59.324187 15185 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-kms
I20251026 15:11:59.324210 15185 mini_ranger_kms.cc:227] Using postgres at 127.14.212.84:49023
2025-10-26 15:11:59,491 [I] DB FLAVOR :POSTGRES
2025-10-26 15:11:59,492 [I] --------- Verifying Ranger DB connection ---------
2025-10-26 15:11:59,492 [I] Checking connection
2025-10-26 15:11:59,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-10-26 15:11:59,837 [I] connection success
2025-10-26 15:11:59,837 [I] --------- Verifying Ranger DB tables ---------
2025-10-26 15:11:59,837 [I] Verifying table ranger_masterkey in database rangerkms
2025-10-26 15:11:59,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-10-26 15:12:00,170 [I] Table ranger_masterkey does not exist in database rangerkms
2025-10-26 15:12:00,171 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-26 15:12:00,171 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-10-26 15:12:00,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.212.84:49023/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-10-26 15:12:00,527 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251026 15:12:01.038537 15185 mini_ranger_kms.cc:310] Created kms service
I20251026 15:12:01.277100 15185 mini_ranger_kms.cc:326] Created kudu user
I20251026 15:12:01.322685 15185 mini_ranger_kms.cc:343] Created rangerkms user
Oct 26 15:12:01 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491519, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
I20251026 15:12:01.736763 15185 mini_ranger_kms.cc:384] Added ranger policy
I20251026 15:12:01.736888 15185 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-kms:/tmp/dist-test-taskXR9jZt/build/release/bin/postgresql.jar:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskXR9jZt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251026 15:12:01.736908 15185 mini_ranger_kms.cc:240] Using host: 127.14.212.84
I20251026 15:12:01.739935 15185 mini_ranger_kms.cc:276] Ranger KMS PID: 27566
I20251026 15:12:01.740049 15185 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.212.84:38555
Oct 26, 2025 3:12:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 26, 2025 3:12:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 26, 2025 3:12:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Oct 26, 2025 3:12:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-kms/ews/webapp] .....
Oct 26, 2025 3:12:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-kms/ews/webapp].
Oct 26, 2025 3:12:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Oct 26, 2025 3:12:03 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-38555"]
Oct 26, 2025 3:12:03 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 26, 2025 3:12:03 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 26, 2025 3:12:03 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 26, 2025 3:12:03 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
I20251026 15:12:03.892989 15185 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.687s user 0.000s sys 0.148s
I20251026 15:12:03.893115 15185 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251026 15:12:03.893167 15185 mini_ranger_kms.cc:401] 127.14.212.84:38555/kms/v1/keys
Oct 26 15:12:03 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761491523, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.212.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Oct 26 15:12:04 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491523, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.212.84@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
Oct 26 15:12:04 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491523, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.212.84@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
Oct 26 15:12:07 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:07.312788 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46393
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:46393
--ranger_config_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.212.84:38555/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:07.393784 27619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:07.393986 27619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:07.394009 27619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:07.395437 27619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:12:07.395485 27619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:07.395499 27619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:07.395512 27619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:12:07.395524 27619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:12:07.397173 27619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.212.84:38555/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:46393
--ranger_config_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:07.397403 27619 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:07.397598 27619 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:07.400467 27624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:07.400647 27619 server_base.cc:1047] running on GCE node
W20251026 15:12:07.400477 27625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:07.400481 27627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:07.400941 27619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:07.401157 27619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:07.402293 27619 hybrid_clock.cc:648] HybridClock initialized: now 1761491527402283 us; error 29 us; skew 500 ppm
Oct 26 15:12:07 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491527, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:07.405164 27619 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:12:07.405725 27619 webserver.cc:492] Webserver started at http://127.14.212.126:33823/ using document root <none> and password file <none>
I20251026 15:12:07.405902 27619 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:07.405936 27619 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:07.406020 27619 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 26 15:12:07 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491527, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
I20251026 15:12:07.449748 27619 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "cbfbb4f36c874611a099cdddfa7b0c7e"
format_stamp: "Formatted at 2025-10-26 15:12:07 on dist-test-slave-8pgz"
server_key: "894c28efb3c60d76c0fe9262992340a6"
server_key_iv: "2d6b4c5072df7086f36d5a7546a933b3"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:07.450125 27619 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "cbfbb4f36c874611a099cdddfa7b0c7e"
format_stamp: "Formatted at 2025-10-26 15:12:07 on dist-test-slave-8pgz"
server_key: "894c28efb3c60d76c0fe9262992340a6"
server_key_iv: "2d6b4c5072df7086f36d5a7546a933b3"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:07.451810 27619 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:12:07.467778 27634 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:07.468050 27619 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:07.468147 27619 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "cbfbb4f36c874611a099cdddfa7b0c7e"
format_stamp: "Formatted at 2025-10-26 15:12:07 on dist-test-slave-8pgz"
server_key: "894c28efb3c60d76c0fe9262992340a6"
server_key_iv: "2d6b4c5072df7086f36d5a7546a933b3"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:07.468238 27619 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:07.494738 27619 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:07.497021 27619 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:07.497141 27619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:07.501833 27619 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:46393
I20251026 15:12:07.502069 27686 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:46393 every 8 connection(s)
I20251026 15:12:07.502260 27619 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:12:07.503019 27687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:07.506124 27687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e: Bootstrap starting.
I20251026 15:12:07.506930 27687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:07.507237 27687 log.cc:826] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:07.508104 27687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e: No bootstrap required, opened a new log
I20251026 15:12:07.508414 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 27619
I20251026 15:12:07.508518 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:12:07.508672 15185 external_mini_cluster.cc:1468] Setting key a36602c599ec275cead4b848b3096a8c
I20251026 15:12:07.509932 27687 raft_consensus.cc:359] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46393 } }
I20251026 15:12:07.510175 27687 raft_consensus.cc:385] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:07.510241 27687 raft_consensus.cc:740] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfbb4f36c874611a099cdddfa7b0c7e, State: Initialized, Role: FOLLOWER
I20251026 15:12:07.510378 27687 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46393 } }
I20251026 15:12:07.510505 27687 raft_consensus.cc:399] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:12:07.510579 27687 raft_consensus.cc:493] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:12:07.510650 27687 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:07.511408 27687 raft_consensus.cc:515] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46393 } }
I20251026 15:12:07.511538 27687 leader_election.cc:304] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbfbb4f36c874611a099cdddfa7b0c7e; no voters:
I20251026 15:12:07.511751 27687 leader_election.cc:290] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:12:07.511795 27692 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:12:07.512044 27692 raft_consensus.cc:697] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 1 LEADER]: Becoming Leader. State: Replica: cbfbb4f36c874611a099cdddfa7b0c7e, State: Running, Role: LEADER
I20251026 15:12:07.512092 27687 sys_catalog.cc:565] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:12:07.512183 27692 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46393 } }
I20251026 15:12:07.512514 27692 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46393 } } }
I20251026 15:12:07.512646 27692 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:07.513842 27687 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251026 15:12:07.513988 27692 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbfbb4f36c874611a099cdddfa7b0c7e. Latest consensus state: current_term: 1 leader_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46393 } } }
I20251026 15:12:07.514132 27692 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [sys.catalog]: This master's current role is: LEADER
Oct 26 15:12:07 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:07.517587 27706 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:12:07.518523 27706 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:12:07.521157 27706 catalog_manager.cc:1357] Generated new cluster ID: eff4bbd6b00541fe91aa1abc96f25999
I20251026 15:12:07.521281 27706 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:12:07.522275 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:07.509151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.14.212.126:46393)
1026 15:12:07.509299 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:07.509311 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:07.509334 (+ 23us) server_negotiation.cc:408] Connection header received
1026 15:12:07.509878 (+ 544us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:07.509883 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:07.509936 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:07.510052 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:07.511147 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:07.512679 (+ 1532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:07.513390 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:07.513647 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:07.516603 (+ 2956us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:07.516687 (+ 84us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:07.516688 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:07.516767 (+ 79us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:07.520033 (+ 3266us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:07.520465 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:07.520467 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:07.520468 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:07.520532 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:07.520786 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:07.520788 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:07.520788 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:07.521083 (+ 295us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:07.521398 (+ 315us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:07.521942 (+ 544us) server_negotiation.cc:300] Negotiation successful
1026 15:12:07.522082 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20251026 15:12:07.536878 27706 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:12:07.537775 27706 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:12:07.544539 27706 catalog_manager.cc:6022] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e: Generated new TSK 0
I20251026 15:12:07.544847 27706 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:08.337493 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.212.84:38555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46393
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:08.456681 27733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:08.456866 27733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:08.456892 27733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:08.458781 27733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:08.458913 27733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:08.459007 27733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:12:08.461571 27733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.212.84:38555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:08.461825 27733 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:08.462072 27733 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:12:08.465164 27733 server_base.cc:1047] running on GCE node
W20251026 15:12:08.465088 27741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:08.465286 27743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:08.465378 27740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:08.465569 27733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:08.465811 27733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:08.466977 27733 hybrid_clock.cc:648] HybridClock initialized: now 1761491528466959 us; error 35 us; skew 500 ppm
Oct 26 15:12:08 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:08.469787 27733 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:12:08.470369 27733 webserver.cc:492] Webserver started at http://127.14.212.65:40993/ using document root <none> and password file <none>
I20251026 15:12:08.470608 27733 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:08.470657 27733 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:08.470775 27733 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 26 15:12:08 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
I20251026 15:12:08.484311 27733 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "477d16c57d414e049744d14d770d73b5"
format_stamp: "Formatted at 2025-10-26 15:12:08 on dist-test-slave-8pgz"
server_key: "a4933919da13150de68f3e9f32e84253"
server_key_iv: "90baf3a05f2071782b053d2c538306dd"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:08.484833 27733 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "477d16c57d414e049744d14d770d73b5"
format_stamp: "Formatted at 2025-10-26 15:12:08 on dist-test-slave-8pgz"
server_key: "a4933919da13150de68f3e9f32e84253"
server_key_iv: "90baf3a05f2071782b053d2c538306dd"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:08.486549 27733 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20251026 15:12:08.506595 27750 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:08.506841 27733 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.000s sys 0.000s
I20251026 15:12:08.506927 27733 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "477d16c57d414e049744d14d770d73b5"
format_stamp: "Formatted at 2025-10-26 15:12:08 on dist-test-slave-8pgz"
server_key: "a4933919da13150de68f3e9f32e84253"
server_key_iv: "90baf3a05f2071782b053d2c538306dd"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:08.507009 27733 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:08.579564 27733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:08.582122 27733 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:08.582283 27733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:08.582535 27733 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:08.582923 27733 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:08.582971 27733 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:08.583004 27733 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:08.583061 27733 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:08.590622 27733 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:40213
I20251026 15:12:08.590683 27865 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:40213 every 8 connection(s)
I20251026 15:12:08.591156 27733 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:12:08.591943 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 27733
I20251026 15:12:08.592049 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:12:08.592227 15185 external_mini_cluster.cc:1468] Setting key 8eb91333f0393f27cca514b518c26879
Oct 26 15:12:08 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
I20251026 15:12:08.608801 27869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:08.591943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:37769 (local address 127.14.212.126:46393)
1026 15:12:08.592049 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:08.592051 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:08.599144 (+ 7093us) server_negotiation.cc:408] Connection header received
1026 15:12:08.599932 (+ 788us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:08.599934 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:08.599951 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:08.600041 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:08.600956 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:08.601838 (+ 882us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:08.603229 (+ 1391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:08.603420 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:08.605310 (+ 1890us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:08.605314 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:08.605314 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:08.605322 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:08.607248 (+ 1926us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:08.607748 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:08.607749 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:08.607749 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:08.607781 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:08.608043 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:08.608044 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:08.608044 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:08.608175 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:08.608255 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:08.608582 (+ 327us) server_negotiation.cc:300] Negotiation successful
1026 15:12:08.608694 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":53,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
I20251026 15:12:08.609428 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:08.591907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46393 (local address 127.14.212.65:37769)
1026 15:12:08.598546 (+ 6639us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:08.598594 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:08.599806 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:08.600122 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:08.600127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:08.600411 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:08.600837 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:08.600841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:08.601948 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:08.601949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:08.602604 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:08.602607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:08.602759 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:08.603736 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:08.603765 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:08.605208 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:08.607346 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:08.607353 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:08.607357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:08.607636 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:08.607868 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:08.607869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:08.607869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:08.607955 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:08.608248 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:08.608252 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:08.608421 (+ 169us) client_negotiation.cc:769] Sending connection context
1026 15:12:08.608544 (+ 123us) client_negotiation.cc:240] Negotiation successful
1026 15:12:08.609202 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6587,"thread_start_us":68,"threads_started":1}
I20251026 15:12:08.610816 27866 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46393
I20251026 15:12:08.610922 27866 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:08.611168 27866 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
I20251026 15:12:08.611768 27651 ts_manager.cc:194] Registered new tserver with Master: 477d16c57d414e049744d14d770d73b5 (127.14.212.65:40213)
I20251026 15:12:08.612792 27651 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:37769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Oct 26 15:12:08 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761491528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:08.658059 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.212.84:38555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46393
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:08.775293 27874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:08.775584 27874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:08.775631 27874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:08.778069 27874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:08.778149 27874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:08.778196 27874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:12:08.780930 27874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.212.84:38555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:08.781180 27874 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:08.781411 27874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:08.784389 27880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:08.784417 27883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:08.784412 27881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:08.784639 27874 server_base.cc:1047] running on GCE node
I20251026 15:12:08.784860 27874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:08.785072 27874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:08.786203 27874 hybrid_clock.cc:648] HybridClock initialized: now 1761491528786190 us; error 34 us; skew 500 ppm
Oct 26 15:12:08 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:08.788872 27874 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:12:08.789371 27874 webserver.cc:492] Webserver started at http://127.14.212.66:37773/ using document root <none> and password file <none>
I20251026 15:12:08.789574 27874 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:08.789616 27874 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:08.789731 27874 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 26 15:12:08 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
I20251026 15:12:08.803722 27874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "7e1e3e9d9f584f878626bbc2c38dfe76"
format_stamp: "Formatted at 2025-10-26 15:12:08 on dist-test-slave-8pgz"
server_key: "0ca462dffe7f9f00b96f1c62ea12e2c2"
server_key_iv: "f3656ea3a6abeb97417dc236e2eeb5e7"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:08.804225 27874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "7e1e3e9d9f584f878626bbc2c38dfe76"
format_stamp: "Formatted at 2025-10-26 15:12:08 on dist-test-slave-8pgz"
server_key: "0ca462dffe7f9f00b96f1c62ea12e2c2"
server_key_iv: "f3656ea3a6abeb97417dc236e2eeb5e7"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:08.805943 27874 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:08.821092 27890 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:08.821336 27874 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251026 15:12:08.821448 27874 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "7e1e3e9d9f584f878626bbc2c38dfe76"
format_stamp: "Formatted at 2025-10-26 15:12:08 on dist-test-slave-8pgz"
server_key: "0ca462dffe7f9f00b96f1c62ea12e2c2"
server_key_iv: "f3656ea3a6abeb97417dc236e2eeb5e7"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:08.821544 27874 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:08.841308 27874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:08.843909 27874 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:08.844108 27874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:08.844355 27874 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:08.844794 27874 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:08.844843 27874 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:08.844874 27874 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:08.844933 27874 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:08.852317 27874 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:34725
I20251026 15:12:08.852782 27874 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:12:08.853312 28003 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:34725 every 8 connection(s)
I20251026 15:12:08.855266 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 27874
I20251026 15:12:08.855371 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:12:08.855618 15185 external_mini_cluster.cc:1468] Setting key 268e48f5d455b52a93453648c038c8e8
Oct 26 15:12:08 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:08.864365 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:08.853562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46393 (local address 127.14.212.66:33063)
1026 15:12:08.853725 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:08.853758 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:08.855044 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:08.855452 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:08.855457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:08.855758 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:08.856182 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:08.856186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:08.857619 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:08.857621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:08.858275 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:08.858277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:08.858460 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:08.859102 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:08.859130 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:08.860722 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:08.862794 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:08.862802 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:08.862805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:08.863084 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:08.863264 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:08.863265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:08.863265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:08.863345 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:08.863660 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:08.863678 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:08.863864 (+ 186us) client_negotiation.cc:769] Sending connection context
1026 15:12:08.863991 (+ 127us) client_negotiation.cc:240] Negotiation successful
1026 15:12:08.864133 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20251026 15:12:08.864913 27869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:08.853574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:33063 (local address 127.14.212.126:46393)
1026 15:12:08.853644 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:08.853646 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:08.853855 (+ 209us) server_negotiation.cc:408] Connection header received
1026 15:12:08.855185 (+ 1330us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:08.855187 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:08.855207 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:08.855244 (+ 37us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:08.856287 (+ 1043us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:08.857084 (+ 797us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:08.858371 (+ 1287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:08.858633 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:08.860831 (+ 2198us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:08.860835 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:08.860836 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:08.860843 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:08.862692 (+ 1849us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:08.863176 (+ 484us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:08.863177 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:08.863178 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:08.863205 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:08.863423 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:08.863425 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:08.863425 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:08.863549 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:08.863626 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:08.864709 (+ 1083us) server_negotiation.cc:300] Negotiation successful
1026 15:12:08.864813 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251026 15:12:08.865682 28004 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46393
I20251026 15:12:08.865833 28004 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:08.866118 28004 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
I20251026 15:12:08.866550 27651 ts_manager.cc:194] Registered new tserver with Master: 7e1e3e9d9f584f878626bbc2c38dfe76 (127.14.212.66:34725)
I20251026 15:12:08.867072 27651 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:33063
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:08.913651 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.212.84:38555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46393
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:09.033039 28011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:09.033227 28011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:09.033269 28011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:09.035185 28011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:09.035254 28011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:09.035295 28011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:12:09.036931 28011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.212.84:38555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:09.037146 28011 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:09.037358 28011 file_cache.cc:492] Constructed file cache file cache with capacity 419430
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
W20251026 15:12:09.040525 28016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:09.040727 28017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:09.040810 28019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:09.041684 28011 server_base.cc:1047] running on GCE node
I20251026 15:12:09.041985 28011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:09.042234 28011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:09.043388 28011 hybrid_clock.cc:648] HybridClock initialized: now 1761491529043373 us; error 28 us; skew 500 ppm
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:09.046654 28011 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:12:09.047323 28011 webserver.cc:492] Webserver started at http://127.14.212.67:46051/ using document root <none> and password file <none>
I20251026 15:12:09.047550 28011 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:09.047602 28011 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:09.047725 28011 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
I20251026 15:12:09.060581 28011 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "68ecf272d8464c20b50993adbb24c4bc"
format_stamp: "Formatted at 2025-10-26 15:12:09 on dist-test-slave-8pgz"
server_key: "5e357347f02dbef3b2af11f1b595d7fb"
server_key_iv: "f26129043766adbfea5b6500e2af46fc"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:09.060956 28011 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "68ecf272d8464c20b50993adbb24c4bc"
format_stamp: "Formatted at 2025-10-26 15:12:09 on dist-test-slave-8pgz"
server_key: "5e357347f02dbef3b2af11f1b595d7fb"
server_key_iv: "f26129043766adbfea5b6500e2af46fc"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:09.063915 28011 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20251026 15:12:09.094604 28026 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:09.094835 28011 fs_manager.cc:730] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20251026 15:12:09.094929 28011 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "68ecf272d8464c20b50993adbb24c4bc"
format_stamp: "Formatted at 2025-10-26 15:12:09 on dist-test-slave-8pgz"
server_key: "5e357347f02dbef3b2af11f1b595d7fb"
server_key_iv: "f26129043766adbfea5b6500e2af46fc"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:09.095049 28011 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:09.104939 28011 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:09.107363 28011 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:09.107510 28011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:09.107702 28011 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:09.108044 28011 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:09.108085 28011 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:09.108122 28011 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:09.108143 28011 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:09.114948 28011 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:46601
I20251026 15:12:09.115337 28011 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:12:09.118052 28139 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:46601 every 8 connection(s)
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:09.124977 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 28011
I20251026 15:12:09.125095 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:12:09.125274 15185 external_mini_cluster.cc:1468] Setting key 741f596dda0794d998853bdb9fbffdd1
I20251026 15:12:09.126323 27869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:09.118706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:40531 (local address 127.14.212.126:46393)
1026 15:12:09.118771 (+ 65us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:09.118774 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:09.118963 (+ 189us) server_negotiation.cc:408] Connection header received
1026 15:12:09.119846 (+ 883us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:09.119847 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:09.119860 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:09.119916 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:09.120551 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.121058 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:09.121591 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.121764 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:09.123487 (+ 1723us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:09.123491 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:09.123491 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:09.123497 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:09.124860 (+ 1363us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:09.125270 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:09.125271 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:09.125272 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:09.125295 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:09.125567 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:09.125568 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:09.125568 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:09.125702 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:09.125744 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:09.126134 (+ 390us) server_negotiation.cc:300] Negotiation successful
1026 15:12:09.126234 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251026 15:12:09.126483 28142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:09.118687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46393 (local address 127.14.212.67:40531)
1026 15:12:09.118841 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:09.118871 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:09.119747 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:09.119973 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:09.119976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:09.120175 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:09.120463 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:09.120465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.121141 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:09.121142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:09.121520 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:09.121521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.121649 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:09.122203 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:09.122220 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:09.123401 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:09.124938 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:09.124943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:09.124946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:09.125209 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:09.125348 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:09.125348 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:09.125349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:09.125430 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:09.125794 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:09.125800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:09.125965 (+ 165us) client_negotiation.cc:769] Sending connection context
1026 15:12:09.126115 (+ 150us) client_negotiation.cc:240] Negotiation successful
1026 15:12:09.126236 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20251026 15:12:09.126871 28140 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46393
I20251026 15:12:09.126967 28140 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:09.127163 28140 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
I20251026 15:12:09.127507 27651 ts_manager.cc:194] Registered new tserver with Master: 68ecf272d8464c20b50993adbb24c4bc (127.14.212.67:46601)
I20251026 15:12:09.127866 27651 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:40531
I20251026 15:12:09.137101 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:12:09.137785 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 27733
I20251026 15:12:09.144595 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 27874
I20251026 15:12:09.150468 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 28011
I20251026 15:12:09.158272 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 27619
I20251026 15:12:09.165424 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46393
--webserver_interface=127.14.212.126
--webserver_port=33823
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:46393
--ranger_config_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.212.84:38555/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.212.84:38555/kms with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:09.249246 28143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:09.249436 28143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:09.249459 28143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:09.250996 28143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:12:09.251055 28143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:09.251070 28143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:09.251083 28143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:12:09.251096 28143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:12:09.252960 28143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.212.84:38555/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:46393
--ranger_config_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=33823
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:09.253206 28143 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:09.253414 28143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:09.256048 28148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:09.256044 28151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:09.256059 28149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:09.256340 28143 server_base.cc:1047] running on GCE node
I20251026 15:12:09.256556 28143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:09.256773 28143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:09.257906 28143 hybrid_clock.cc:648] HybridClock initialized: now 1761491529257894 us; error 29 us; skew 500 ppm
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:09.260432 28143 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:12:09.260988 28143 webserver.cc:492] Webserver started at http://127.14.212.126:33823/ using document root <none> and password file <none>
I20251026 15:12:09.261169 28143 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:09.261216 28143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
I20251026 15:12:09.311756 28143 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:09.312531 28159 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:09.312736 28143 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:09.312832 28143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "cbfbb4f36c874611a099cdddfa7b0c7e"
format_stamp: "Formatted at 2025-10-26 15:12:07 on dist-test-slave-8pgz"
server_key: "894c28efb3c60d76c0fe9262992340a6"
server_key_iv: "2d6b4c5072df7086f36d5a7546a933b3"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:09.313251 28143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:09.329214 28143 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:09.331526 28143 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:09.331657 28143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:09.335783 28143 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:46393
I20251026 15:12:09.336011 28211 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:46393 every 8 connection(s)
I20251026 15:12:09.336205 28143 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:12:09.337319 28212 sys_catalog.cc:263] Verifying existing consensus state
I20251026 15:12:09.337949 28212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e: Bootstrap starting.
I20251026 15:12:09.341506 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 28143
I20251026 15:12:09.341835 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.212.84:38555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:40213
--local_ip_for_outbound_sockets=127.14.212.65
--tserver_master_addrs=127.14.212.126:46393
--webserver_port=40993
--webserver_interface=127.14.212.65
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.212.84:38555/kms with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:12:09.346768 28212 log.cc:826] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:09.348300 28212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 15:12:09.348558 28212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e: Bootstrap complete.
I20251026 15:12:09.350539 28212 raft_consensus.cc:359] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46393 } }
I20251026 15:12:09.350754 28212 raft_consensus.cc:740] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfbb4f36c874611a099cdddfa7b0c7e, State: Initialized, Role: FOLLOWER
I20251026 15:12:09.350883 28212 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46393 } }
I20251026 15:12:09.350950 28212 raft_consensus.cc:399] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:12:09.350998 28212 raft_consensus.cc:493] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:12:09.351043 28212 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 1 FOLLOWER]: Advancing to term 2
I20251026 15:12:09.351897 28212 raft_consensus.cc:515] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46393 } }
I20251026 15:12:09.351995 28212 leader_election.cc:304] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbfbb4f36c874611a099cdddfa7b0c7e; no voters:
I20251026 15:12:09.352156 28212 leader_election.cc:290] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [CANDIDATE]: Term 2 election: Requested vote from peers
I20251026 15:12:09.352208 28216 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 2 FOLLOWER]: Leader election won for term 2
I20251026 15:12:09.352310 28216 raft_consensus.cc:697] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [term 2 LEADER]: Becoming Leader. State: Replica: cbfbb4f36c874611a099cdddfa7b0c7e, State: Running, Role: LEADER
I20251026 15:12:09.352393 28212 sys_catalog.cc:565] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:12:09.352386 28216 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46393 } }
I20251026 15:12:09.352967 28217 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46393 } } }
I20251026 15:12:09.353106 28217 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:09.353369 28226 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:12:09.353669 28218 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbfbb4f36c874611a099cdddfa7b0c7e. Latest consensus state: current_term: 2 leader_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfbb4f36c874611a099cdddfa7b0c7e" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46393 } } }
I20251026 15:12:09.353753 28218 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:09.353855 28226 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:12:09.353989 28226 catalog_manager.cc:1269] Loaded cluster ID: eff4bbd6b00541fe91aa1abc96f25999
I20251026 15:12:09.354018 28226 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:12:09.354208 28212 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251026 15:12:09.355289 28226 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:12:09.355479 28226 catalog_manager.cc:6033] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e: Loaded TSK: 0
I20251026 15:12:09.355669 28226 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251026 15:12:09.450520 28214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:09.450778 28214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:09.450819 28214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:09.453198 28214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:09.453267 28214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:09.453306 28214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:12:09.455996 28214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.212.84:38555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:40213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=40993
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:09.456254 28214 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:09.456446 28214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:09.460479 28252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:09.460935 28249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:09.461203 28250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:09.461504 28214 server_base.cc:1047] running on GCE node
I20251026 15:12:09.461736 28214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:09.461958 28214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:09.463107 28214 hybrid_clock.cc:648] HybridClock initialized: now 1761491529463081 us; error 39 us; skew 500 ppm
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:09.466354 28214 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:12:09.467204 28214 webserver.cc:492] Webserver started at http://127.14.212.65:40993/ using document root <none> and password file <none>
I20251026 15:12:09.467427 28214 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:09.467491 28214 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
I20251026 15:12:09.490818 28214 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20251026 15:12:09.503504 28260 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:09.503854 28214 fs_manager.cc:730] Time spent opening block manager: real 0.013s user 0.000s sys 0.000s
I20251026 15:12:09.503968 28214 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "477d16c57d414e049744d14d770d73b5"
format_stamp: "Formatted at 2025-10-26 15:12:08 on dist-test-slave-8pgz"
server_key: "a4933919da13150de68f3e9f32e84253"
server_key_iv: "90baf3a05f2071782b053d2c538306dd"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:09.504354 28214 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:09.538338 28214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:09.548460 28214 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:09.548640 28214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:09.548913 28214 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:09.549300 28214 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:09.549351 28214 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:09.549418 28214 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:09.549503 28214 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:09.557057 28214 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:40213
I20251026 15:12:09.557551 28214 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:12:09.558660 28373 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:40213 every 8 connection(s)
I20251026 15:12:09.560418 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 28214
I20251026 15:12:09.560761 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.212.84:38555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:34725
--local_ip_for_outbound_sockets=127.14.212.66
--tserver_master_addrs=127.14.212.126:46393
--webserver_port=37773
--webserver_interface=127.14.212.66
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.212.84:38555/kms with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:09.578454 28376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:09.559206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46393 (local address 127.14.212.65:60521)
1026 15:12:09.559384 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:09.559421 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:09.560647 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:09.561122 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:09.561127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:09.561422 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:09.561843 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:09.561847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.564432 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:09.564433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:09.564983 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:09.564987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.565139 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:09.567524 (+ 2385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:09.567555 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:09.569342 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:09.574035 (+ 4693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:09.574045 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:09.574049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:09.574368 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:09.574606 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:09.574607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:09.574608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:09.574680 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:09.575106 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:09.575112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:09.575257 (+ 145us) client_negotiation.cc:769] Sending connection context
1026 15:12:09.575384 (+ 127us) client_negotiation.cc:240] Negotiation successful
1026 15:12:09.577954 (+ 2570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20251026 15:12:09.579664 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:09.559547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:60521 (local address 127.14.212.126:46393)
1026 15:12:09.559880 (+ 333us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:09.559891 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:09.559916 (+ 25us) server_negotiation.cc:408] Connection header received
1026 15:12:09.560869 (+ 953us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:09.560876 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:09.560929 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:09.561064 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:09.561983 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.564290 (+ 2307us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:09.568186 (+ 3896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.568489 (+ 303us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:09.569540 (+ 1051us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:09.569551 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:09.569551 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:09.569576 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:09.573900 (+ 4324us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:09.574483 (+ 583us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:09.574486 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:09.574487 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:09.574545 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:09.574737 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:09.574738 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:09.574738 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:09.574974 (+ 236us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:09.575071 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:09.579323 (+ 4252us) server_negotiation.cc:300] Negotiation successful
1026 15:12:09.579468 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":76,"threads_started":1}
I20251026 15:12:09.580747 28374 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46393
I20251026 15:12:09.580880 28374 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:09.581146 28374 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
W20251026 15:12:09.581928 28374 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:46393 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20251026 15:12:09.582000 28374 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:09.582122 28374 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
I20251026 15:12:09.582877 28374 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:09.582991 28374 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
W20251026 15:12:09.583257 28374 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251026 15:12:09.666716 28378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:09.666968 28378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:09.667011 28378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:09.669296 28378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:09.669360 28378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:09.669400 28378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:12:09.673709 28378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.212.84:38555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:34725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=37773
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:09.673969 28378 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:09.674183 28378 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:09.677408 28384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:09.677852 28383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:09.678288 28386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:09.678561 28378 server_base.cc:1047] running on GCE node
I20251026 15:12:09.678745 28378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:09.678958 28378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:09.680109 28378 hybrid_clock.cc:648] HybridClock initialized: now 1761491529680085 us; error 40 us; skew 500 ppm
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:09.682865 28378 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:12:09.683398 28378 webserver.cc:492] Webserver started at http://127.14.212.66:37773/ using document root <none> and password file <none>
I20251026 15:12:09.683604 28378 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:09.683650 28378 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
I20251026 15:12:09.696540 28378 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:09.697304 28394 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:09.697549 28378 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:09.697630 28378 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "7e1e3e9d9f584f878626bbc2c38dfe76"
format_stamp: "Formatted at 2025-10-26 15:12:08 on dist-test-slave-8pgz"
server_key: "0ca462dffe7f9f00b96f1c62ea12e2c2"
server_key_iv: "f3656ea3a6abeb97417dc236e2eeb5e7"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:09.698014 28378 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:09.716007 28378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:09.718314 28378 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:09.718492 28378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:09.718704 28378 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:09.719075 28378 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:09.719130 28378 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:09.719172 28378 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:09.719197 28378 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:09.726984 28378 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:34725
I20251026 15:12:09.727471 28378 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:12:09.730607 28507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:34725 every 8 connection(s)
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:09.737013 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 28378
I20251026 15:12:09.737318 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.212.84:38555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:46601
--local_ip_for_outbound_sockets=127.14.212.67
--tserver_master_addrs=127.14.212.126:46393
--webserver_port=46051
--webserver_interface=127.14.212.67
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.212.84:38555/kms with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:12:09.738538 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:09.728267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:36539 (local address 127.14.212.126:46393)
1026 15:12:09.728319 (+ 52us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:09.728322 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:09.728571 (+ 249us) server_negotiation.cc:408] Connection header received
1026 15:12:09.729921 (+ 1350us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:09.729923 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:09.729940 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:09.729996 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:09.731010 (+ 1014us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.731565 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:09.732443 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.732638 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:09.735051 (+ 2413us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:09.735057 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:09.735058 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:09.735067 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:09.736789 (+ 1722us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:09.737325 (+ 536us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:09.737328 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:09.737328 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:09.737376 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:09.737626 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:09.737627 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:09.737627 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:09.737755 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:09.737806 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:09.738278 (+ 472us) server_negotiation.cc:300] Negotiation successful
1026 15:12:09.738442 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251026 15:12:09.739140 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:09.728272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46393 (local address 127.14.212.66:36539)
1026 15:12:09.728429 (+ 157us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:09.728468 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:09.729783 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:09.730086 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:09.730091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:09.730442 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:09.730897 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:09.730903 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.731662 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:09.731663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:09.732306 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:09.732309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.732777 (+ 468us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:09.733485 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:09.733515 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:09.734948 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:09.736901 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:09.736910 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:09.736914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:09.737220 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:09.737453 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:09.737454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:09.737454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:09.737553 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:09.737864 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:09.737870 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:09.738071 (+ 201us) client_negotiation.cc:769] Sending connection context
1026 15:12:09.738705 (+ 634us) client_negotiation.cc:240] Negotiation successful
1026 15:12:09.738851 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20251026 15:12:09.739746 28508 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46393
I20251026 15:12:09.739861 28508 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:09.740120 28508 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
W20251026 15:12:09.740707 28508 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:46393 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20251026 15:12:09.740778 28508 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:09.740882 28508 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
I20251026 15:12:09.741195 28508 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:09.741300 28508 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
W20251026 15:12:09.741597 28508 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251026 15:12:09.840178 28511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:09.840428 28511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:09.840466 28511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:09.842887 28511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:09.842959 28511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:09.843003 28511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:12:09.845696 28511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.212.84:38555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:46601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=46051
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:09.845964 28511 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:09.846179 28511 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:09.849246 28517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:09.849256 28519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:09.849570 28516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:09.850021 28511 server_base.cc:1047] running on GCE node
I20251026 15:12:09.850204 28511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:09.850481 28511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:09.851656 28511 hybrid_clock.cc:648] HybridClock initialized: now 1761491529851639 us; error 34 us; skew 500 ppm
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:09.854985 28511 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:12:09.856062 28511 webserver.cc:492] Webserver started at http://127.14.212.67:46051/ using document root <none> and password file <none>
I20251026 15:12:09.856376 28511 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:09.856505 28511 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
I20251026 15:12:09.873827 28511 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20251026 15:12:09.878752 28527 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:09.878985 28511 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s
I20251026 15:12:09.879089 28511 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "68ecf272d8464c20b50993adbb24c4bc"
format_stamp: "Formatted at 2025-10-26 15:12:09 on dist-test-slave-8pgz"
server_key: "5e357347f02dbef3b2af11f1b595d7fb"
server_key_iv: "f26129043766adbfea5b6500e2af46fc"
server_key_version: "kuduclusterkey@0"
I20251026 15:12:09.879453 28511 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:09.903280 28511 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:09.905865 28511 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:09.906005 28511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:09.906209 28511 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:09.906580 28511 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:09.906631 28511 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:09.906672 28511 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:09.906697 28511 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:09.913972 28511 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:46601
I20251026 15:12:09.914108 28640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:46601 every 8 connection(s)
I20251026 15:12:09.914451 28511 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
Oct 26 15:12:09 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:09.923826 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 28511
I20251026 15:12:09.926635 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:09.915251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:48459 (local address 127.14.212.126:46393)
1026 15:12:09.915298 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:09.915300 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:09.915572 (+ 272us) server_negotiation.cc:408] Connection header received
1026 15:12:09.916789 (+ 1217us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:09.916791 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:09.916805 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:09.916886 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:09.918748 (+ 1862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.919503 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:09.920320 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.920517 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:09.922921 (+ 2404us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:09.922926 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:09.922927 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:09.922935 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:09.924793 (+ 1858us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:09.925534 (+ 741us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:09.925535 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:09.925536 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:09.925565 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:09.925835 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:09.925837 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:09.925837 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:09.925964 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:09.926044 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:09.926416 (+ 372us) server_negotiation.cc:300] Negotiation successful
1026 15:12:09.926523 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251026 15:12:09.926779 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:09.915246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46393 (local address 127.14.212.67:48459)
1026 15:12:09.915397 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:09.915431 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:09.916646 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:09.917685 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:09.917690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:09.918073 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:09.918622 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:09.918627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.919615 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:09.919616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:09.920215 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:09.920218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.920378 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:09.921094 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:09.921121 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:09.922802 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:09.924895 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:09.924904 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:09.924908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:09.925443 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:09.925664 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:09.925665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:09.925665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:09.925764 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:09.926038 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:09.926044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:09.926224 (+ 180us) client_negotiation.cc:769] Sending connection context
1026 15:12:09.926353 (+ 129us) client_negotiation.cc:240] Negotiation successful
1026 15:12:09.926541 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20251026 15:12:09.927435 28641 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46393
I20251026 15:12:09.927551 28641 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:09.927812 28641 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
W20251026 15:12:09.928498 28641 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:46393 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20251026 15:12:09.928573 28641 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:09.928668 28641 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
I20251026 15:12:09.929055 28641 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:09.929155 28641 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
W20251026 15:12:09.929518 28641 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251026 15:12:09.950578 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:09.925071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.14.212.126:46393)
1026 15:12:09.925586 (+ 515us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:09.925588 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:09.934739 (+ 9151us) server_negotiation.cc:408] Connection header received
1026 15:12:09.934776 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:09.934777 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:09.934793 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:09.934855 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:09.935970 (+ 1115us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.936797 (+ 827us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:09.945399 (+ 8602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:09.945604 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:09.946984 (+ 1380us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:09.946991 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:09.946991 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:09.946999 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:09.949337 (+ 2338us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:09.949766 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:09.949768 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:09.949768 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:09.949794 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:09.949954 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:09.949955 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:09.949955 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:09.950078 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:09.950234 (+ 156us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:09.950343 (+ 109us) server_negotiation.cc:300] Negotiation successful
1026 15:12:09.950468 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":90,"threads_started":1}
Oct 26 15:12:10 dist-test-slave-8pgz krb5kdc[27148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761491530, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:10.583434 28374 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:10.583650 28374 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
I20251026 15:12:10.584091 28176 ts_manager.cc:194] Registered new tserver with Master: 477d16c57d414e049744d14d770d73b5 (127.14.212.65:40213)
I20251026 15:12:10.585240 28176 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:60521
I20251026 15:12:10.741775 28508 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:10.741981 28508 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
I20251026 15:12:10.742307 28176 ts_manager.cc:194] Registered new tserver with Master: 7e1e3e9d9f584f878626bbc2c38dfe76 (127.14.212.66:34725)
I20251026 15:12:10.742739 28176 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:36539
Oct 26 15:12:10 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491530, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
I20251026 15:12:10.929721 28641 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:10.929939 28641 heartbeater.cc:507] Master 127.14.212.126:46393 requested a full tablet report, sending...
I20251026 15:12:10.930308 28176 ts_manager.cc:194] Registered new tserver with Master: 68ecf272d8464c20b50993adbb24c4bc (127.14.212.67:46601)
I20251026 15:12:10.930893 28176 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:48459
I20251026 15:12:10.935051 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:12:10.943499 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:10.936251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.14.212.126:46393)
1026 15:12:10.936348 (+ 97us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:10.936350 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:10.936363 (+ 13us) server_negotiation.cc:408] Connection header received
1026 15:12:10.936386 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:10.936387 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:10.936411 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:10.936484 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:10.937328 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.938150 (+ 822us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:10.939210 (+ 1060us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.939394 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:10.940478 (+ 1084us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:10.940482 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:10.940482 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:10.940489 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:10.942429 (+ 1940us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:10.942802 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:10.942803 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:10.942804 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:10.942828 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:10.942969 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:10.942970 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:10.942970 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:10.943079 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:10.943177 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:10.943287 (+ 110us) server_negotiation.cc:300] Negotiation successful
1026 15:12:10.943377 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":53,"threads_started":1}
I20251026 15:12:10.945649 28176 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:12:10.947572 28176 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:12:10.953877 28668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:10.950484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:40213 (local address 127.0.0.1:36852)
1026 15:12:10.950706 (+ 222us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:10.950716 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:10.950788 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:10.951506 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:10.951509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:10.951511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:10.951773 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:10.951775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.952684 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:10.952686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:10.953592 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:10.953594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.953672 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:10.953709 (+ 37us) client_negotiation.cc:769] Sending connection context
1026 15:12:10.953755 (+ 46us) client_negotiation.cc:240] Negotiation successful
1026 15:12:10.953795 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":69,"threads_started":1}
I20251026 15:12:10.954895 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:10.950571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.14.212.65:40213)
1026 15:12:10.950904 (+ 333us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:10.950907 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:10.950915 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:12:10.951020 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:10.951023 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:10.951044 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:10.951857 (+ 813us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:10.951939 (+ 82us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.952599 (+ 660us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:10.954116 (+ 1517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.954710 (+ 594us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:10.954768 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:10.954798 (+ 30us) server_negotiation.cc:300] Negotiation successful
1026 15:12:10.954829 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":46,"threads_started":1}
I20251026 15:12:10.955444 28669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:10.950360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59632 (local address 127.14.212.67:46601)
1026 15:12:10.950541 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:10.950547 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:10.950740 (+ 193us) server_negotiation.cc:408] Connection header received
1026 15:12:10.951472 (+ 732us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:10.951476 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:10.951498 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:10.951552 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:10.952004 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.953083 (+ 1079us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:10.954577 (+ 1494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.955112 (+ 535us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:10.955160 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:10.955317 (+ 157us) server_negotiation.cc:300] Negotiation successful
1026 15:12:10.955361 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20251026 15:12:10.955451 28308 tablet_service.cc:1505] Processing CreateTablet for tablet 182e42fbfc0340d89661b59dbc591d20 (DEFAULT_TABLE table=test-table [id=e7fdadfc2e3946ee9bdc8b55beeb3b05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:12:10.955806 28308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 182e42fbfc0340d89661b59dbc591d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:10.955941 28672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:10.950634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:34725 (local address 127.0.0.1:46232)
1026 15:12:10.951146 (+ 512us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:10.951155 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:10.951204 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:10.952691 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:10.952692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:10.952694 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:10.952926 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:10.952927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.954494 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:10.954495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:10.955645 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:10.955649 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.955719 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:10.955723 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:12:10.955754 (+ 31us) client_negotiation.cc:240] Negotiation successful
1026 15:12:10.955795 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":93,"threads_started":1}
I20251026 15:12:10.956854 28673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:10.950854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.14.212.66:34725)
1026 15:12:10.950990 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:10.950994 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:10.951257 (+ 263us) server_negotiation.cc:408] Connection header received
1026 15:12:10.951297 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:10.951299 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:10.951317 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:10.951377 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:10.953010 (+ 1633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.954371 (+ 1361us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:10.956148 (+ 1777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.956674 (+ 526us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:10.956723 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:10.956754 (+ 31us) server_negotiation.cc:300] Negotiation successful
1026 15:12:10.956788 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20251026 15:12:10.956017 28670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:10.950237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:46601 (local address 127.0.0.1:59632)
1026 15:12:10.950624 (+ 387us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:10.950652 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:10.950788 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:10.951685 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:10.951686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:10.951689 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:10.951907 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:10.951910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.953465 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:10.953466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:10.954459 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:10.954462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:10.955199 (+ 737us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:10.955203 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:12:10.955864 (+ 661us) client_negotiation.cc:240] Negotiation successful
1026 15:12:10.955911 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":87,"threads_started":1}
I20251026 15:12:10.957557 28442 tablet_service.cc:1505] Processing CreateTablet for tablet 182e42fbfc0340d89661b59dbc591d20 (DEFAULT_TABLE table=test-table [id=e7fdadfc2e3946ee9bdc8b55beeb3b05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:12:10.957870 28442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 182e42fbfc0340d89661b59dbc591d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:10.957769 28575 tablet_service.cc:1505] Processing CreateTablet for tablet 182e42fbfc0340d89661b59dbc591d20 (DEFAULT_TABLE table=test-table [id=e7fdadfc2e3946ee9bdc8b55beeb3b05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:12:10.958094 28575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 182e42fbfc0340d89661b59dbc591d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:10.959872 28674 tablet_bootstrap.cc:492] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5: Bootstrap starting.
I20251026 15:12:10.960469 28674 tablet_bootstrap.cc:654] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:10.960769 28674 log.cc:826] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:10.961074 28676 tablet_bootstrap.cc:492] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc: Bootstrap starting.
I20251026 15:12:10.961424 28675 tablet_bootstrap.cc:492] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76: Bootstrap starting.
I20251026 15:12:10.961592 28674 tablet_bootstrap.cc:492] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5: No bootstrap required, opened a new log
I20251026 15:12:10.961674 28674 ts_tablet_manager.cc:1403] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:12:10.961966 28675 tablet_bootstrap.cc:654] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:10.962244 28675 log.cc:826] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:10.962987 28675 tablet_bootstrap.cc:492] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76: No bootstrap required, opened a new log
I20251026 15:12:10.963058 28675 ts_tablet_manager.cc:1403] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 15:12:10.963788 28674 raft_consensus.cc:359] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ecf272d8464c20b50993adbb24c4bc" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46601 } } peers { permanent_uuid: "477d16c57d414e049744d14d770d73b5" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 40213 } } peers { permanent_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34725 } }
I20251026 15:12:10.963964 28674 raft_consensus.cc:385] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:10.964010 28674 raft_consensus.cc:740] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477d16c57d414e049744d14d770d73b5, State: Initialized, Role: FOLLOWER
I20251026 15:12:10.964151 28674 consensus_queue.cc:260] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ecf272d8464c20b50993adbb24c4bc" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46601 } } peers { permanent_uuid: "477d16c57d414e049744d14d770d73b5" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 40213 } } peers { permanent_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34725 } }
I20251026 15:12:10.964401 28674 ts_tablet_manager.cc:1434] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251026 15:12:10.964524 28374 heartbeater.cc:499] Master 127.14.212.126:46393 was elected leader, sending a full tablet report...
I20251026 15:12:10.965062 28675 raft_consensus.cc:359] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ecf272d8464c20b50993adbb24c4bc" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46601 } } peers { permanent_uuid: "477d16c57d414e049744d14d770d73b5" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 40213 } } peers { permanent_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34725 } }
I20251026 15:12:10.965205 28675 raft_consensus.cc:385] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:10.965242 28675 raft_consensus.cc:740] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e1e3e9d9f584f878626bbc2c38dfe76, State: Initialized, Role: FOLLOWER
I20251026 15:12:10.965205 28676 tablet_bootstrap.cc:654] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:10.965333 28675 consensus_queue.cc:260] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ecf272d8464c20b50993adbb24c4bc" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46601 } } peers { permanent_uuid: "477d16c57d414e049744d14d770d73b5" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 40213 } } peers { permanent_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34725 } }
I20251026 15:12:10.965590 28675 ts_tablet_manager.cc:1434] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251026 15:12:10.965723 28508 heartbeater.cc:499] Master 127.14.212.126:46393 was elected leader, sending a full tablet report...
I20251026 15:12:10.966236 28676 log.cc:826] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:10.967285 28676 tablet_bootstrap.cc:492] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc: No bootstrap required, opened a new log
I20251026 15:12:10.967365 28676 ts_tablet_manager.cc:1403] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20251026 15:12:10.969439 28676 raft_consensus.cc:359] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ecf272d8464c20b50993adbb24c4bc" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46601 } } peers { permanent_uuid: "477d16c57d414e049744d14d770d73b5" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 40213 } } peers { permanent_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34725 } }
I20251026 15:12:10.969606 28676 raft_consensus.cc:385] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:10.969635 28676 raft_consensus.cc:740] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ecf272d8464c20b50993adbb24c4bc, State: Initialized, Role: FOLLOWER
I20251026 15:12:10.969766 28676 consensus_queue.cc:260] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ecf272d8464c20b50993adbb24c4bc" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46601 } } peers { permanent_uuid: "477d16c57d414e049744d14d770d73b5" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 40213 } } peers { permanent_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34725 } }
I20251026 15:12:10.970041 28676 ts_tablet_manager.cc:1434] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251026 15:12:10.970072 28641 heartbeater.cc:499] Master 127.14.212.126:46393 was elected leader, sending a full tablet report...
W20251026 15:12:10.978722 28509 tablet.cc:2378] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:12:11.028061 28680 raft_consensus.cc:493] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:12:11.028199 28680 raft_consensus.cc:515] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ecf272d8464c20b50993adbb24c4bc" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46601 } } peers { permanent_uuid: "477d16c57d414e049744d14d770d73b5" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 40213 } } peers { permanent_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34725 } }
I20251026 15:12:11.028541 28680 leader_election.cc:290] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68ecf272d8464c20b50993adbb24c4bc (127.14.212.67:46601), 477d16c57d414e049744d14d770d73b5 (127.14.212.65:40213)
I20251026 15:12:11.032960 28669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:11.029360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:56841 (local address 127.14.212.67:46601)
1026 15:12:11.029454 (+ 94us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:11.029456 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:11.029636 (+ 180us) server_negotiation.cc:408] Connection header received
1026 15:12:11.029747 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:11.029748 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:11.029763 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:11.029792 (+ 29us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:11.030249 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:11.030999 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:11.032223 (+ 1224us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:11.032708 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:11.032718 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:11.032840 (+ 122us) server_negotiation.cc:300] Negotiation successful
1026 15:12:11.032884 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251026 15:12:11.033416 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:11.028812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:46601 (local address 127.14.212.66:56841)
1026 15:12:11.029554 (+ 742us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:11.029565 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:11.029668 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:11.029872 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:11.029873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:11.029875 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:11.030157 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:11.030158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:11.031112 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:11.031113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:11.032117 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:11.032120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:11.032750 (+ 630us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:11.032754 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:12:11.033289 (+ 535us) client_negotiation.cc:240] Negotiation successful
1026 15:12:11.033335 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":111,"threads_started":1}
I20251026 15:12:11.034217 28595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "182e42fbfc0340d89661b59dbc591d20" candidate_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ecf272d8464c20b50993adbb24c4bc" is_pre_election: true
I20251026 15:12:11.034411 28595 raft_consensus.cc:2468] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e1e3e9d9f584f878626bbc2c38dfe76 in term 0.
I20251026 15:12:11.034694 28397 leader_election.cc:304] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68ecf272d8464c20b50993adbb24c4bc, 7e1e3e9d9f584f878626bbc2c38dfe76; no voters:
I20251026 15:12:11.034842 28680 raft_consensus.cc:2804] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:12:11.034907 28680 raft_consensus.cc:493] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:12:11.034940 28680 raft_consensus.cc:3060] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:11.035617 28684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:11.029024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:40213 (local address 127.14.212.66:57205)
1026 15:12:11.031187 (+ 2163us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:11.031200 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:11.031309 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:11.031523 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:11.031524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:11.031526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:11.031823 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:11.031825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:11.032689 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:11.032690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:11.033521 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:11.033525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:11.035403 (+ 1878us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:11.035407 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:12:11.035465 (+ 58us) client_negotiation.cc:240] Negotiation successful
1026 15:12:11.035508 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2140,"thread_start_us":68,"threads_started":1}
I20251026 15:12:11.035779 28680 raft_consensus.cc:515] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ecf272d8464c20b50993adbb24c4bc" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46601 } } peers { permanent_uuid: "477d16c57d414e049744d14d770d73b5" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 40213 } } peers { permanent_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34725 } }
I20251026 15:12:11.035917 28680 leader_election.cc:290] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [CANDIDATE]: Term 1 election: Requested vote from peers 68ecf272d8464c20b50993adbb24c4bc (127.14.212.67:46601), 477d16c57d414e049744d14d770d73b5 (127.14.212.65:40213)
I20251026 15:12:11.036142 28595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "182e42fbfc0340d89661b59dbc591d20" candidate_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ecf272d8464c20b50993adbb24c4bc"
I20251026 15:12:11.036233 28595 raft_consensus.cc:3060] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:11.036254 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:11.029408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:57205 (local address 127.14.212.65:40213)
1026 15:12:11.029488 (+ 80us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:11.029490 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:11.031275 (+ 1785us) server_negotiation.cc:408] Connection header received
1026 15:12:11.031391 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:11.031393 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:11.031408 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:11.031440 (+ 32us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:11.031926 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:11.032572 (+ 646us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:11.033625 (+ 1053us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:11.034064 (+ 439us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:11.034073 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:11.036135 (+ 2062us) server_negotiation.cc:300] Negotiation successful
1026 15:12:11.036187 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251026 15:12:11.036514 28324 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "182e42fbfc0340d89661b59dbc591d20" candidate_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477d16c57d414e049744d14d770d73b5" is_pre_election: true
I20251026 15:12:11.036511 28325 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "182e42fbfc0340d89661b59dbc591d20" candidate_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477d16c57d414e049744d14d770d73b5"
I20251026 15:12:11.036693 28324 raft_consensus.cc:2468] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e1e3e9d9f584f878626bbc2c38dfe76 in term 0.
I20251026 15:12:11.037410 28595 raft_consensus.cc:2468] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e1e3e9d9f584f878626bbc2c38dfe76 in term 1.
I20251026 15:12:11.037585 28397 leader_election.cc:304] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68ecf272d8464c20b50993adbb24c4bc, 7e1e3e9d9f584f878626bbc2c38dfe76; no voters: 477d16c57d414e049744d14d770d73b5
I20251026 15:12:11.037695 28680 raft_consensus.cc:2804] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:12:11.037832 28680 raft_consensus.cc:697] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [term 1 LEADER]: Becoming Leader. State: Replica: 7e1e3e9d9f584f878626bbc2c38dfe76, State: Running, Role: LEADER
I20251026 15:12:11.037945 28680 consensus_queue.cc:237] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ecf272d8464c20b50993adbb24c4bc" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46601 } } peers { permanent_uuid: "477d16c57d414e049744d14d770d73b5" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 40213 } } peers { permanent_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34725 } }
I20251026 15:12:11.038832 28176 catalog_manager.cc:5649] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e1e3e9d9f584f878626bbc2c38dfe76 (127.14.212.66). New cstate: current_term: 1 leader_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ecf272d8464c20b50993adbb24c4bc" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "477d16c57d414e049744d14d770d73b5" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 40213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e1e3e9d9f584f878626bbc2c38dfe76" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34725 } health_report { overall_health: HEALTHY } } }
Oct 26 15:12:11 dist-test-slave-8pgz krb5kdc[27148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491530, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for HTTP/127.14.212.84@KRBTEST.COM
I20251026 15:12:11.050920 28673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:11.048026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46234 (local address 127.14.212.66:34725)
1026 15:12:11.048066 (+ 40us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:11.048068 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:11.048075 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:12:11.048117 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:11.048118 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:11.048139 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:11.048190 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:12:11.048563 (+ 373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:11.049294 (+ 731us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:11.050184 (+ 890us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:11.050413 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:11.050470 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:12:11.050604 (+ 134us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:12:11.050757 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:11.050796 (+ 39us) server_negotiation.cc:300] Negotiation successful
1026 15:12:11.050832 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20251026 15:12:11.060815 28375 tablet.cc:2378] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:12:11.063480 28324 raft_consensus.cc:3060] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:11.064517 28324 raft_consensus.cc:1275] T 182e42fbfc0340d89661b59dbc591d20 P 477d16c57d414e049744d14d770d73b5 [term 1 FOLLOWER]: Refusing update from remote peer 7e1e3e9d9f584f878626bbc2c38dfe76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:12:11.064814 28595 raft_consensus.cc:1275] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc [term 1 FOLLOWER]: Refusing update from remote peer 7e1e3e9d9f584f878626bbc2c38dfe76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:12:11.065084 28686 consensus_queue.cc:1048] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "477d16c57d414e049744d14d770d73b5" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 40213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:12:11.065272 28680 consensus_queue.cc:1048] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ecf272d8464c20b50993adbb24c4bc" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:12:11.071942 28689 mvcc.cc:204] Tried to move back new op lower bound from 7215069311233212416 to 7215069311132712960. Current Snapshot: MvccSnapshot[applied={T|T < 7215069311233212416}]
I20251026 15:12:11.072358 28690 mvcc.cc:204] Tried to move back new op lower bound from 7215069311233212416 to 7215069311132712960. Current Snapshot: MvccSnapshot[applied={T|T < 7215069311233212416}]
I20251026 15:12:11.076998 28176 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54700:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:12:11.077116 28176 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54700:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:12:11.077875 28176 catalog_manager.cc:5936] T 00000000000000000000000000000000 P cbfbb4f36c874611a099cdddfa7b0c7e: Sending DeleteTablet for 3 replicas of tablet 182e42fbfc0340d89661b59dbc591d20
I20251026 15:12:11.078341 28308 tablet_service.cc:1552] Processing DeleteTablet for tablet 182e42fbfc0340d89661b59dbc591d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:11 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:36852
I20251026 15:12:11.078375 28575 tablet_service.cc:1552] Processing DeleteTablet for tablet 182e42fbfc0340d89661b59dbc591d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:11 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:59632
I20251026 15:12:11.078667 28699 tablet_replica.cc:333] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc: stopping tablet replica
I20251026 15:12:11.078706 28442 tablet_service.cc:1552] Processing DeleteTablet for tablet 182e42fbfc0340d89661b59dbc591d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:11 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:46232
I20251026 15:12:11.078781 28699 raft_consensus.cc:2243] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:12:11.078891 28699 raft_consensus.cc:2272] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:12:11.079100 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 28214
I20251026 15:12:11.081012 28699 ts_tablet_manager.cc:1916] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:12:11.086561 28700 tablet_replica.cc:333] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76: stopping tablet replica
I20251026 15:12:11.086680 28700 raft_consensus.cc:2243] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:12:11.086825 28700 raft_consensus.cc:2272] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:12:11.087337 28700 ts_tablet_manager.cc:1916] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:12:11.087157 28699 ts_tablet_manager.cc:1929] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:12:11.087704 28699 log.cc:1199] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/182e42fbfc0340d89661b59dbc591d20
I20251026 15:12:11.089630 28699 ts_tablet_manager.cc:1950] T 182e42fbfc0340d89661b59dbc591d20 P 68ecf272d8464c20b50993adbb24c4bc: Deleting consensus metadata
I20251026 15:12:11.089637 28700 ts_tablet_manager.cc:1929] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:12:11.089859 28700 log.cc:1199] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/182e42fbfc0340d89661b59dbc591d20
I20251026 15:12:11.090231 28700 ts_tablet_manager.cc:1950] T 182e42fbfc0340d89661b59dbc591d20 P 7e1e3e9d9f584f878626bbc2c38dfe76: Deleting consensus metadata
I20251026 15:12:11.090759 28162 catalog_manager.cc:4985] TS 7e1e3e9d9f584f878626bbc2c38dfe76 (127.14.212.66:34725): tablet 182e42fbfc0340d89661b59dbc591d20 (table test-table [id=e7fdadfc2e3946ee9bdc8b55beeb3b05]) successfully deleted
W20251026 15:12:11.091828 28162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:40213 (error 108)
I20251026 15:12:11.091955 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 28378
I20251026 15:12:11.093712 28162 catalog_manager.cc:4985] TS 68ecf272d8464c20b50993adbb24c4bc (127.14.212.67:46601): tablet 182e42fbfc0340d89661b59dbc591d20 (table test-table [id=e7fdadfc2e3946ee9bdc8b55beeb3b05]) successfully deleted
I20251026 15:12:11.094348 28670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:11.094200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:40213 (local address 127.0.0.1:36866)
1026 15:12:11.094264 (+ 64us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:11.094306 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20251026 15:12:11.094494 28162 catalog_manager.cc:4712] TS 477d16c57d414e049744d14d770d73b5 (127.14.212.65:40213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 182e42fbfc0340d89661b59dbc591d20: Network error: Client connection negotiation failed: client connection to 127.14.212.65:40213: connect: Connection refused (error 111)
I20251026 15:12:11.098526 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 28511
I20251026 15:12:11.105122 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 28143
I20251026 15:12:11.114439 15185 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20251026 15:12:11.469584 15185 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20251026 15:12:11.469714 15185 mini_ranger.cc:66] Stopping Ranger...
I20251026 15:12:11.815371 15185 mini_ranger.cc:68] Stopped Ranger
2025-10-26 15:12:11.815 UTC [27179] LOG: received smart shutdown request
2025-10-26 15:12:11.819 UTC [27179] LOG: background worker "logical replication launcher" (PID 27187) exited with exit code 1
2025-10-26 15:12:11.819 UTC [27182] LOG: shutting down
2025-10-26 15:12:11.821 UTC [27182] LOG: checkpoint starting: shutdown immediate
2025-10-26 15:12:12.795 UTC [27182] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.036 s, sync=0.932 s, total=0.976 s; sync files=1108, longest=0.006 s, average=0.001 s; distance=12074 kB, estimate=12074 kB; lsn=0/2045130, redo lsn=0/2045130
2025-10-26 15:12:12.804 UTC [27179] LOG: database system is shut down
2025-10-26T15:12:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53051 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:12:12 dist-test-slave-8pgz krb5kdc[28711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:12:12 dist-test-slave-8pgz krb5kdc[28711](info): set up 2 sockets
Oct 26 15:12:12 dist-test-slave-8pgz krb5kdc[28711](info): commencing operation
krb5kdc: starting...
W20251026 15:12:14.931877 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.995s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:12:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:12:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:15.071579 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:43941
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:37793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:43941,127.14.212.125:46661,127.14.212.124:44573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:15.152351 28727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:15.152556 28727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:15.152592 28727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:15.154067 28727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:12:15.154124 28727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:15.154146 28727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:15.154170 28727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:12:15.154186 28727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:12:15.155874 28727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:37793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:43941,127.14.212.125:46661,127.14.212.124:44573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:43941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:15.156150 28727 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:15.156342 28727 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:15.158994 28735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:15.159000 28733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:15.159065 28727 server_base.cc:1047] running on GCE node
W20251026 15:12:15.159153 28732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:15.159531 28727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:15.159754 28727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:15.160899 28727 hybrid_clock.cc:648] HybridClock initialized: now 1761491535160881 us; error 34 us; skew 500 ppm
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:15.163484 28727 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:12:15.164032 28727 webserver.cc:492] Webserver started at http://127.14.212.126:41479/ using document root <none> and password file <none>
I20251026 15:12:15.164224 28727 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:15.164259 28727 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:15.164371 28727 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:12:15.165345 28727 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "2092e6eb477547099dbcb8bf9efd0adb"
format_stamp: "Formatted at 2025-10-26 15:12:15 on dist-test-slave-8pgz"
server_key: "ec878aa4a326c49c979c4b6c72a18bd0"
server_key_iv: "48a09b920ae56b28ab0851ca4c84f54e"
server_key_version: "encryptionkey@0"
I20251026 15:12:15.165647 28727 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "2092e6eb477547099dbcb8bf9efd0adb"
format_stamp: "Formatted at 2025-10-26 15:12:15 on dist-test-slave-8pgz"
server_key: "ec878aa4a326c49c979c4b6c72a18bd0"
server_key_iv: "48a09b920ae56b28ab0851ca4c84f54e"
server_key_version: "encryptionkey@0"
I20251026 15:12:15.167109 28727 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:12:15.167950 28742 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:15.168226 28727 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:15.168298 28727 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "2092e6eb477547099dbcb8bf9efd0adb"
format_stamp: "Formatted at 2025-10-26 15:12:15 on dist-test-slave-8pgz"
server_key: "ec878aa4a326c49c979c4b6c72a18bd0"
server_key_iv: "48a09b920ae56b28ab0851ca4c84f54e"
server_key_version: "encryptionkey@0"
I20251026 15:12:15.168408 28727 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:15.184566 28727 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:15.190634 28727 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:15.190771 28727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:15.195057 28727 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:43941
I20251026 15:12:15.195091 28794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:43941 every 8 connection(s)
I20251026 15:12:15.195436 28727 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:12:15.196041 28795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:15.196473 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 28727
I20251026 15:12:15.196584 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:12:15.196758 15185 external_mini_cluster.cc:1468] Setting key c6ada08e890ceeb6bdb66146588ba1fa
I20251026 15:12:15.198802 28795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.212.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.125@KRBTEST.COM" created.
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:15.212114 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.199596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43941 (local address 127.0.0.1:41384)
1026 15:12:15.199829 (+ 233us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.199863 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:15.200415 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:15.200722 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:15.200726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:15.201191 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:15.201584 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.201591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.202696 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.202698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:15.203281 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.203284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.203433 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.204078 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:15.204107 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:15.208459 (+ 4352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:15.210793 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.210800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.210803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.210980 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.211164 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.211165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.211166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.211217 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.211590 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:15.211595 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:15.211660 (+ 65us) client_negotiation.cc:769] Sending connection context
1026 15:12:15.211746 (+ 86us) client_negotiation.cc:240] Negotiation successful
1026 15:12:15.211965 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":90,"thread_start_us":142,"threads_started":1}
I20251026 15:12:15.212127 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.199626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.14.212.126:43941)
1026 15:12:15.199848 (+ 222us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.199860 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.199951 (+ 91us) server_negotiation.cc:408] Connection header received
1026 15:12:15.200590 (+ 639us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.200593 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.200611 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.200700 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.201722 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.202579 (+ 857us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.203389 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.203562 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.208582 (+ 5020us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.208593 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.208594 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.208602 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.210696 (+ 2094us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.211061 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.211064 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.211064 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.211106 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.211271 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.211272 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.211273 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.211514 (+ 241us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.211607 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.211774 (+ 167us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.211893 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":137,"threads_started":1}
I20251026 15:12:15.212924 28795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:12:15.213454 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.213273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:46661 (local address 127.0.0.1:41962)
1026 15:12:15.213369 (+ 96us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.213421 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:46661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20251026 15:12:15.213598 28746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.125:46661: connect: Connection refused (error 111)
I20251026 15:12:15.214075 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.213974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:46661 (local address 127.0.0.1:41970)
1026 15:12:15.214023 (+ 49us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.214052 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:46661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20251026 15:12:15.214186 28795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.125:46661: Network error: Client connection negotiation failed: client connection to 127.14.212.125:46661: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:15.249933 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.125:46661
--webserver_interface=127.14.212.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:37793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:43941,127.14.212.125:46661,127.14.212.124:44573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:12:15.263396 28795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } attempt: 1
I20251026 15:12:15.263922 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.263777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:46661 (local address 127.0.0.1:41986)
1026 15:12:15.263856 (+ 79us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.263889 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:46661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20251026 15:12:15.264314 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.264214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:46661 (local address 127.0.0.1:42002)
1026 15:12:15.264268 (+ 54us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.264292 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:46661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20251026 15:12:15.264425 28795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.125:46661: Network error: Client connection negotiation failed: client connection to 127.14.212.125:46661: connect: Connection refused (error 111)
W20251026 15:12:15.331565 28804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:15.331763 28804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:15.331787 28804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251026 15:12:15.332659 28795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } attempt: 2
I20251026 15:12:15.333165 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.333025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:46661 (local address 127.0.0.1:42008)
1026 15:12:15.333085 (+ 60us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.333132 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:46661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20251026 15:12:15.333495 28804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20251026 15:12:15.333527 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.333444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:46661 (local address 127.0.0.1:42022)
1026 15:12:15.333473 (+ 29us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.333501 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:46661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20251026 15:12:15.333551 28804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:15.333578 28804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:15.333606 28804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:12:15.333626 28804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251026 15:12:15.333663 28795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.125:46661: Network error: Client connection negotiation failed: client connection to 127.14.212.125:46661: connect: Connection refused (error 111)
I20251026 15:12:15.335513 28804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:37793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:43941,127.14.212.125:46661,127.14.212.124:44573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.125:46661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.212.125
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:15.335801 28804 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:15.335994 28804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:15.338708 28809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:15.338703 28810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:15.338722 28812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:15.338747 28804 server_base.cc:1047] running on GCE node
I20251026 15:12:15.339115 28804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:15.339321 28804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:15.340456 28804 hybrid_clock.cc:648] HybridClock initialized: now 1761491535340448 us; error 32 us; skew 500 ppm
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:15.342803 28804 init.cc:377] Logged in from keytab as kudu/127.14.212.125@KRBTEST.COM (short username kudu)
I20251026 15:12:15.343377 28804 webserver.cc:492] Webserver started at http://127.14.212.125:38621/ using document root <none> and password file <none>
I20251026 15:12:15.343573 28804 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:15.343616 28804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:15.343729 28804 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:12:15.344739 28804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/data/instance:
uuid: "c3549ca467d34532abc6ecc29a893511"
format_stamp: "Formatted at 2025-10-26 15:12:15 on dist-test-slave-8pgz"
server_key: "df25c9156fb4d865a8af448dea77d97f"
server_key_iv: "dad92fa7cad2c9d40091498611ea3908"
server_key_version: "encryptionkey@0"
I20251026 15:12:15.345054 28804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/wal/instance:
uuid: "c3549ca467d34532abc6ecc29a893511"
format_stamp: "Formatted at 2025-10-26 15:12:15 on dist-test-slave-8pgz"
server_key: "df25c9156fb4d865a8af448dea77d97f"
server_key_iv: "dad92fa7cad2c9d40091498611ea3908"
server_key_version: "encryptionkey@0"
I20251026 15:12:15.346458 28804 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251026 15:12:15.347325 28819 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:15.347600 28804 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:12:15.347679 28804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/wal
uuid: "c3549ca467d34532abc6ecc29a893511"
format_stamp: "Formatted at 2025-10-26 15:12:15 on dist-test-slave-8pgz"
server_key: "df25c9156fb4d865a8af448dea77d97f"
server_key_iv: "dad92fa7cad2c9d40091498611ea3908"
server_key_version: "encryptionkey@0"
I20251026 15:12:15.347754 28804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:15.361999 28804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:15.364187 28804 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:15.364395 28804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:15.368583 28804 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.125:46661
I20251026 15:12:15.368620 28871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.125:46661 every 8 connection(s)
I20251026 15:12:15.368943 28804 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/data/info.pb
I20251026 15:12:15.369585 28872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:15.371465 28872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:12:15.374843 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 28804
I20251026 15:12:15.374933 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-1/wal/instance
I20251026 15:12:15.375064 15185 external_mini_cluster.cc:1468] Setting key f50fe33f459ef24f82856ea7c05df355
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.125@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:15.380800 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.372169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.14.212.126:43941)
1026 15:12:15.372234 (+ 65us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.372237 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.372495 (+ 258us) server_negotiation.cc:408] Connection header received
1026 15:12:15.372945 (+ 450us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.372946 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.372963 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.373044 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.374226 (+ 1182us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.374788 (+ 562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.375450 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.375647 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.377794 (+ 2147us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.377798 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.377799 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.377806 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.379564 (+ 1758us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.379967 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.379969 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.379969 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.379995 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.380183 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.380184 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.380185 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.380284 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.380346 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.380609 (+ 263us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.380709 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251026 15:12:15.380945 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.372137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43941 (local address 127.0.0.1:41390)
1026 15:12:15.372289 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.372328 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:15.372825 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:15.373132 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:15.373137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:15.373758 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:15.374109 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.374114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.374881 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.374882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:15.375348 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.375350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.375505 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.376237 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:15.376259 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:15.377693 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:15.379669 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.379677 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.379680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.379891 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.380051 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.380052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.380053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.380127 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.380339 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:15.380344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:15.380486 (+ 142us) client_negotiation.cc:769] Sending connection context
1026 15:12:15.380587 (+ 101us) client_negotiation.cc:240] Negotiation successful
1026 15:12:15.380734 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20251026 15:12:15.381433 28872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.212.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.124@KRBTEST.COM" created.
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.125@KRBTEST.COM for kudu/127.14.212.125@KRBTEST.COM
I20251026 15:12:15.389827 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.381682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:46661 (local address 127.0.0.1:42026)
1026 15:12:15.381726 (+ 44us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.381737 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:15.381777 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:15.382047 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:15.382048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:15.382189 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:15.382312 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.382314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.383148 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.383149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:15.383480 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.383482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.383548 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.384048 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:15.384052 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:15.386274 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:15.388780 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.388781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.388782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.388928 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.389148 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.389148 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.389149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.389171 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.389537 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:15.389538 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:15.389586 (+ 48us) client_negotiation.cc:769] Sending connection context
1026 15:12:15.389659 (+ 73us) client_negotiation.cc:240] Negotiation successful
1026 15:12:15.389751 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20251026 15:12:15.389998 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.381766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.14.212.125:46661)
1026 15:12:15.381919 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.381926 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.381940 (+ 14us) server_negotiation.cc:408] Connection header received
1026 15:12:15.381960 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.381963 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.381980 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.382037 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.382387 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.383072 (+ 685us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.383550 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.383763 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.386451 (+ 2688us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.386468 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.386468 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.386475 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.388673 (+ 2198us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.389019 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.389023 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.389024 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.389082 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.389208 (+ 126us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.389208 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.389209 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.389372 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.389589 (+ 217us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.389744 (+ 155us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.389852 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":105,"threads_started":1}
I20251026 15:12:15.390723 28872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:12:15.391095 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.390949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49758)
1026 15:12:15.391001 (+ 52us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.391060 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20251026 15:12:15.391268 28821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
I20251026 15:12:15.391698 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.391592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49770)
1026 15:12:15.391637 (+ 45us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.391676 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20251026 15:12:15.391870 28872 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:44573: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.124@KRBTEST.COM" created.
I20251026 15:12:15.424866 28795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:15.425884 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.124:44573
--webserver_interface=127.14.212.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:37793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:43941,127.14.212.125:46661,127.14.212.124:44573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for kudu/127.14.212.125@KRBTEST.COM
I20251026 15:12:15.432497 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.425274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:46661 (local address 127.0.0.1:42038)
1026 15:12:15.425364 (+ 90us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.425385 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:15.425433 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:15.425625 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:15.425627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:15.425790 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:15.425946 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.425948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.426743 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.426744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:15.427090 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.427091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.427162 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.427766 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:15.427771 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:15.429239 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:15.431515 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.431516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.431517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.431688 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.431861 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.431861 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.431862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.431884 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.432157 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:15.432158 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:15.432208 (+ 50us) client_negotiation.cc:769] Sending connection context
1026 15:12:15.432268 (+ 60us) client_negotiation.cc:240] Negotiation successful
1026 15:12:15.432369 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20251026 15:12:15.432583 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.425354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.14.212.125:46661)
1026 15:12:15.425451 (+ 97us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.425453 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.425466 (+ 13us) server_negotiation.cc:408] Connection header received
1026 15:12:15.425498 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.425500 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.425516 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.425599 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.426107 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.426664 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.427165 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.427370 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.429333 (+ 1963us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.429337 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.429338 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.429346 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.431403 (+ 2057us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.431765 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.431766 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.431766 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.431797 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.431935 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.431936 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.431937 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.432084 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.432159 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.432312 (+ 153us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.432449 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251026 15:12:15.433637 28795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:12:15.434093 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.433953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49784)
1026 15:12:15.434013 (+ 60us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.434053 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20251026 15:12:15.434490 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.434365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49794)
1026 15:12:15.434434 (+ 69us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.434465 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20251026 15:12:15.434608 28795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:44573: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
I20251026 15:12:15.441097 28872 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } attempt: 1
I20251026 15:12:15.441608 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.441474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49796)
1026 15:12:15.441532 (+ 58us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.441575 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20251026 15:12:15.441979 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.441886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49802)
1026 15:12:15.441931 (+ 45us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.441958 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20251026 15:12:15.442107 28872 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:44573: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
I20251026 15:12:15.465803 28795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } attempt: 1
I20251026 15:12:15.466418 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.466221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49816)
1026 15:12:15.466294 (+ 73us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.466349 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20251026 15:12:15.466794 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.466690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49826)
1026 15:12:15.466741 (+ 51us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.466773 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20251026 15:12:15.466889 28795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:44573: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
I20251026 15:12:15.510267 28872 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } attempt: 2
I20251026 15:12:15.510800 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.510633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49840)
1026 15:12:15.510708 (+ 75us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.510759 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20251026 15:12:15.511197 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.511095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49854)
1026 15:12:15.511145 (+ 50us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.511176 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20251026 15:12:15.511307 28872 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:44573: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
W20251026 15:12:15.511411 28881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:15.511698 28881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:15.511726 28881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:15.513283 28881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:12:15.513330 28881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:15.513345 28881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:15.513360 28881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:12:15.513372 28881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:12:15.515242 28881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:37793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:43941,127.14.212.125:46661,127.14.212.124:44573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.124:44573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.212.124
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:15.515497 28881 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:15.515699 28881 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:15.518164 28887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:15.518276 28886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:15.518196 28889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:15.518507 28881 server_base.cc:1047] running on GCE node
I20251026 15:12:15.518698 28881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:15.518905 28881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:15.520051 28881 hybrid_clock.cc:648] HybridClock initialized: now 1761491535520037 us; error 29 us; skew 500 ppm
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:15.522748 28881 init.cc:377] Logged in from keytab as kudu/127.14.212.124@KRBTEST.COM (short username kudu)
I20251026 15:12:15.523319 28881 webserver.cc:492] Webserver started at http://127.14.212.124:38511/ using document root <none> and password file <none>
I20251026 15:12:15.523504 28881 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:15.523540 28881 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:15.523643 28881 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:12:15.524755 28881 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/data/instance:
uuid: "e650e0beb1b44540aec2430062dc9b1a"
format_stamp: "Formatted at 2025-10-26 15:12:15 on dist-test-slave-8pgz"
server_key: "b397b9456815313bad772c4da082bfe1"
server_key_iv: "7273b45e8f0332048bca93a645374734"
server_key_version: "encryptionkey@0"
I20251026 15:12:15.525089 28881 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/wal/instance:
uuid: "e650e0beb1b44540aec2430062dc9b1a"
format_stamp: "Formatted at 2025-10-26 15:12:15 on dist-test-slave-8pgz"
server_key: "b397b9456815313bad772c4da082bfe1"
server_key_iv: "7273b45e8f0332048bca93a645374734"
server_key_version: "encryptionkey@0"
I20251026 15:12:15.526340 28881 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251026 15:12:15.527159 28896 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:15.527437 28881 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:15.527504 28881 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/wal
uuid: "e650e0beb1b44540aec2430062dc9b1a"
format_stamp: "Formatted at 2025-10-26 15:12:15 on dist-test-slave-8pgz"
server_key: "b397b9456815313bad772c4da082bfe1"
server_key_iv: "7273b45e8f0332048bca93a645374734"
server_key_version: "encryptionkey@0"
I20251026 15:12:15.527575 28881 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:15.541620 28881 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:15.542033 28795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } attempt: 2
I20251026 15:12:15.542465 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.542298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49858)
1026 15:12:15.542369 (+ 71us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.542421 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20251026 15:12:15.542855 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.542761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49860)
1026 15:12:15.542806 (+ 45us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.542829 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20251026 15:12:15.542936 28795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:44573: Network error: Client connection negotiation failed: client connection to 127.14.212.124:44573: connect: Connection refused (error 111)
I20251026 15:12:15.544124 28881 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:15.544276 28881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:15.548478 28881 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.124:44573
I20251026 15:12:15.548540 28948 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.124:44573 every 8 connection(s)
I20251026 15:12:15.548851 28881 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/data/info.pb
I20251026 15:12:15.549546 28949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:15.550987 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 28881
I20251026 15:12:15.551072 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761491252308956-15185-0/minicluster-data/master-2/wal/instance
I20251026 15:12:15.551239 15185 external_mini_cluster.cc:1468] Setting key 99bd936f423f1b11875d06678aa895cb
I20251026 15:12:15.551565 28949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } has no permanent_uuid. Determining permanent_uuid...
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.124@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.124@KRBTEST.COM
I20251026 15:12:15.560609 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.552279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.14.212.126:43941)
1026 15:12:15.552345 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.552347 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.552475 (+ 128us) server_negotiation.cc:408] Connection header received
1026 15:12:15.552502 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.552504 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.552516 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.552572 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.553501 (+ 929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.554231 (+ 730us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.555015 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.555157 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.557656 (+ 2499us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.557661 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.557663 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.557673 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.559317 (+ 1644us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.559733 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.559734 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.559735 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.559760 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.559934 (+ 174us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.559935 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.559936 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.560038 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.560109 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.560426 (+ 317us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.560524 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251026 15:12:15.560758 28953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.552176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43941 (local address 127.0.0.1:41406)
1026 15:12:15.552373 (+ 197us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.552394 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:15.552457 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:15.552586 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:15.552588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:15.553020 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:15.553406 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.553410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.554327 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.554328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:15.554873 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.554876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.555017 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.555823 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:15.555861 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:15.557552 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:15.559403 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.559410 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.559413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.559659 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.559815 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.559815 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.559816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.559880 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.560136 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:15.560141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:15.560270 (+ 129us) client_negotiation.cc:769] Sending connection context
1026 15:12:15.560401 (+ 131us) client_negotiation.cc:240] Negotiation successful
1026 15:12:15.560545 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20251026 15:12:15.561008 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.551786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49868 (local address 127.14.212.124:44573)
1026 15:12:15.551964 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.551975 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.551998 (+ 23us) server_negotiation.cc:408] Connection header received
1026 15:12:15.552350 (+ 352us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.552358 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.552409 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.552533 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.553505 (+ 972us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.554504 (+ 999us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.555263 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.555488 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.558133 (+ 2645us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.558143 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.558144 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.558151 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.560007 (+ 1856us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.560315 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.560318 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.560318 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.560347 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.560486 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.560487 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.560487 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.560598 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.560731 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.560835 (+ 104us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.560925 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":77,"threads_started":1}
I20251026 15:12:15.561676 28949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } has no permanent_uuid. Determining permanent_uuid...
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.124@KRBTEST.COM for kudu/127.14.212.125@KRBTEST.COM
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:15.568452 28953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.561944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:46661 (local address 127.0.0.1:42040)
1026 15:12:15.562026 (+ 82us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.562039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:15.562088 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:15.562268 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:15.562270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:15.562544 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:15.562757 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.562760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.563447 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.563448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:15.563824 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.563826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.563906 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.564442 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:15.564447 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:15.565839 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:15.567606 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.567607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.567607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.567750 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.567901 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.567902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.567902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.567929 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.568132 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:15.568132 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:15.568178 (+ 46us) client_negotiation.cc:769] Sending connection context
1026 15:12:15.568237 (+ 59us) client_negotiation.cc:240] Negotiation successful
1026 15:12:15.568347 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20251026 15:12:15.568451 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.562063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.14.212.125:46661)
1026 15:12:15.562133 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.562135 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.562144 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:12:15.562173 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.562174 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.562185 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.562240 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.562849 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.563313 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.563924 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.564086 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.565944 (+ 1858us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.565948 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.565949 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.565955 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.567487 (+ 1532us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.567818 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.567820 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.567820 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.567846 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.567979 (+ 133us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.567980 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.567980 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.568077 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.568138 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.568263 (+ 125us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.568356 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251026 15:12:15.568866 28949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:12:15.569226 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.562277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.14.212.126:43941)
1026 15:12:15.562312 (+ 35us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.562314 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.562351 (+ 37us) server_negotiation.cc:408] Connection header received
1026 15:12:15.562520 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.562521 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.562530 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.562590 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.563255 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.563736 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.564337 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.564564 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.566611 (+ 2047us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.566617 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.566617 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.566624 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.568071 (+ 1447us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.568383 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.568384 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.568385 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.568410 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.568565 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.568567 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.568567 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.568699 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.568822 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.569008 (+ 186us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.569111 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.124@KRBTEST.COM for kudu/127.14.212.124@KRBTEST.COM
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.125@KRBTEST.COM
I20251026 15:12:15.575805 28953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.569126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49882)
1026 15:12:15.569181 (+ 55us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.569192 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:15.569226 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:15.569342 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:15.569343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:15.569508 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:15.569675 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.569677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.570359 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.570360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:15.570796 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.570798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.570871 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.571446 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:15.571450 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:15.572797 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:15.574797 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.574798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.574798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.574974 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.575162 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.575163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.575164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.575202 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.575545 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:15.575545 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:15.575584 (+ 39us) client_negotiation.cc:769] Sending connection context
1026 15:12:15.575652 (+ 68us) client_negotiation.cc:240] Negotiation successful
1026 15:12:15.575732 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20251026 15:12:15.575932 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.569192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49882 (local address 127.14.212.124:44573)
1026 15:12:15.569232 (+ 40us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.569234 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.569246 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:12:15.569270 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.569270 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.569278 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.569331 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.569733 (+ 402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.570285 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.570897 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.571038 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.572902 (+ 1864us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.572906 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.572906 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.572914 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.574723 (+ 1809us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.575051 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.575052 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.575052 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.575082 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.575249 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.575250 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.575250 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.575453 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.575555 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.575698 (+ 143us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.575824 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":22}
I20251026 15:12:15.577158 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.569960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.14.212.125:46661)
1026 15:12:15.569997 (+ 37us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.569999 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.570006 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:12:15.570025 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.570026 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.570035 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.570089 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.570685 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.571321 (+ 636us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.571851 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.571979 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.574149 (+ 2170us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.574154 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.574154 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.574162 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.576135 (+ 1973us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.576504 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.576506 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.576506 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.576532 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.576698 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.576699 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.576699 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.576801 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.576922 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.577018 (+ 96us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.577092 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20251026 15:12:15.577749 28949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a: Bootstrap starting.
I20251026 15:12:15.578732 28949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:15.578843 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:12:15.579064 28949 log.cc:826] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:15.579943 28949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a: No bootstrap required, opened a new log
I20251026 15:12:15.581590 28949 raft_consensus.cc:359] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } }
I20251026 15:12:15.581714 28949 raft_consensus.cc:385] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:15.581745 28949 raft_consensus.cc:740] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e650e0beb1b44540aec2430062dc9b1a, State: Initialized, Role: FOLLOWER
I20251026 15:12:15.581851 28949 consensus_queue.cc:260] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } }
I20251026 15:12:15.582124 28956 sys_catalog.cc:455] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } } }
I20251026 15:12:15.582171 28949 sys_catalog.cc:565] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:12:15.582218 28956 sys_catalog.cc:458] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [sys.catalog]: This master's current role is: FOLLOWER
W20251026 15:12:15.584189 28967 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:12:15.584318 28967 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:12:15.584640 28881 master_runner.cc:186] Error getting master registration for 127.14.212.126:43941: OK, instance_id { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" instance_seqno: 1761491535193805 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251026 15:12:15.586684 28881 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.124@KRBTEST.COM
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.125@KRBTEST.COM
I20251026 15:12:15.602440 28872 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } attempt: 3
I20251026 15:12:15.605619 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.595923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.14.212.124:44573)
1026 15:12:15.595964 (+ 41us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.595966 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.596073 (+ 107us) server_negotiation.cc:408] Connection header received
1026 15:12:15.596173 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.596175 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.596185 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.596215 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.597154 (+ 939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.597808 (+ 654us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.598503 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.598676 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.601557 (+ 2881us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.601562 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.601563 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.601572 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.604316 (+ 2744us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.604706 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.604707 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.604708 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.604731 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.604877 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.604878 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.604878 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.605025 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.605114 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.605424 (+ 310us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.605507 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251026 15:12:15.606096 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.595694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.14.212.126:43941)
1026 15:12:15.595791 (+ 97us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.595793 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.595937 (+ 144us) server_negotiation.cc:408] Connection header received
1026 15:12:15.596036 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.596037 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.596048 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.596079 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.597076 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.597780 (+ 704us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.598593 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.598789 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.602666 (+ 3877us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.602670 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.602671 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.602679 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.604870 (+ 2191us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.605225 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.605226 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.605227 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.605255 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.605411 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.605412 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.605412 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.605524 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.605622 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.605931 (+ 309us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.606018 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20251026 15:12:15.607344 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.595647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.14.212.125:46661)
1026 15:12:15.595740 (+ 93us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.595742 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.595757 (+ 15us) server_negotiation.cc:408] Connection header received
1026 15:12:15.595862 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.595864 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.595877 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.595910 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.597027 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.597706 (+ 679us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.598503 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.598676 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.602687 (+ 4011us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.602691 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.602691 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.602698 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.605677 (+ 2979us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.606652 (+ 975us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.606653 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.606653 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.606678 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.606827 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.606828 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.606828 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.606922 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.606985 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.607173 (+ 188us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.607251 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.125@KRBTEST.COM for kudu/127.14.212.124@KRBTEST.COM
I20251026 15:12:15.612658 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.602952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.14.212.124:44573)
1026 15:12:15.603104 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.603105 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.603704 (+ 599us) server_negotiation.cc:408] Connection header received
1026 15:12:15.603731 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.603732 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.603744 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.604288 (+ 544us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.604344 (+ 56us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.606958 (+ 2614us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.607755 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.607942 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.609763 (+ 1821us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.609767 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.609767 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.609773 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.611739 (+ 1966us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.612094 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.612095 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.612095 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.612116 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.612249 (+ 133us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.612249 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.612249 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.612327 (+ 78us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.612371 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.612505 (+ 134us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.612572 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20251026 15:12:15.612690 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.602751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49902)
1026 15:12:15.602841 (+ 90us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.603469 (+ 628us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:15.603521 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:15.603844 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:15.603845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:15.604007 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:15.604204 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.604207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.607048 (+ 2841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.607049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:15.607666 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.607669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.607749 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.608284 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:15.608289 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:15.609570 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:15.611826 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.611827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.611827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.612013 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.612168 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.612168 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.612169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.612197 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.612374 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:15.612375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:15.612420 (+ 45us) client_negotiation.cc:769] Sending connection context
1026 15:12:15.612490 (+ 70us) client_negotiation.cc:240] Negotiation successful
1026 15:12:15.612594 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20251026 15:12:15.613945 28872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511: Bootstrap starting.
I20251026 15:12:15.614825 28872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:15.615106 28872 log.cc:826] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:15.615837 28872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511: No bootstrap required, opened a new log
I20251026 15:12:15.617192 28872 raft_consensus.cc:359] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } }
I20251026 15:12:15.617296 28872 raft_consensus.cc:385] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:15.617331 28872 raft_consensus.cc:740] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3549ca467d34532abc6ecc29a893511, State: Initialized, Role: FOLLOWER
I20251026 15:12:15.617444 28872 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } }
I20251026 15:12:15.617704 28979 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } } }
I20251026 15:12:15.617743 28872 sys_catalog.cc:565] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:12:15.618485 28979 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [sys.catalog]: This master's current role is: FOLLOWER
W20251026 15:12:15.619748 28990 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:12:15.619802 28990 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:12:15.619951 28804 master_runner.cc:186] Error getting master registration for 127.14.212.126:43941: OK, instance_id { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" instance_seqno: 1761491535193805 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251026 15:12:15.621657 28804 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251026 15:12:15.642086 28795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } attempt: 3
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for kudu/127.14.212.124@KRBTEST.COM
I20251026 15:12:15.650151 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.642479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.14.212.124:44573)
1026 15:12:15.642548 (+ 69us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:15.642550 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:15.642563 (+ 13us) server_negotiation.cc:408] Connection header received
1026 15:12:15.642601 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:15.642602 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:15.642615 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:15.642682 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:15.643196 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.643968 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.644726 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.644919 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.647130 (+ 2211us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:15.647136 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:15.647136 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:15.647144 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:15.649056 (+ 1912us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.649443 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.649444 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.649444 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.649471 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:15.649627 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:15.649627 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:15.649628 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:15.649746 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:15.649814 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:15.649962 (+ 148us) server_negotiation.cc:300] Negotiation successful
1026 15:12:15.650047 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251026 15:12:15.650149 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:15.642437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:44573 (local address 127.0.0.1:49904)
1026 15:12:15.642490 (+ 53us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:15.642501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:15.642543 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:15.642700 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:15.642701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:15.642894 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:15.643104 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.643106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.644062 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:15.644064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:15.644633 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:15.644635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:15.644712 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:15.645405 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:15.645411 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:15.647033 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:15.649157 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.649159 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.649159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.649358 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.649531 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:15.649532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:15.649533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:15.649564 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:15.649808 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:15.649809 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:15.649861 (+ 52us) client_negotiation.cc:769] Sending connection context
1026 15:12:15.649935 (+ 74us) client_negotiation.cc:240] Negotiation successful
1026 15:12:15.650042 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20251026 15:12:15.652266 28795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb: Bootstrap starting.
I20251026 15:12:15.653275 28795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:15.653586 28795 log.cc:826] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:15.654577 28795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb: No bootstrap required, opened a new log
I20251026 15:12:15.656277 28795 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } }
I20251026 15:12:15.656432 28795 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:15.656469 28795 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2092e6eb477547099dbcb8bf9efd0adb, State: Initialized, Role: FOLLOWER
I20251026 15:12:15.656610 28795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } }
I20251026 15:12:15.656961 28795 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:12:15.656927 28993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } } }
I20251026 15:12:15.657121 28993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [sys.catalog]: This master's current role is: FOLLOWER
W20251026 15:12:15.659030 29004 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:12:15.659157 29004 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:12:15.661110 28727 master_runner.cc:263] No leader master found from master 2092e6eb477547099dbcb8bf9efd0adb
I20251026 15:12:15.661170 28727 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251026 15:12:15.691591 28748 tablet.cc:2378] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:12:15.698053 28979 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:12:15.698215 28979 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } }
I20251026 15:12:15.698701 28979 leader_election.cc:290] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2092e6eb477547099dbcb8bf9efd0adb (127.14.212.126:43941), e650e0beb1b44540aec2430062dc9b1a (127.14.212.124:44573)
I20251026 15:12:15.699146 28923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3549ca467d34532abc6ecc29a893511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e650e0beb1b44540aec2430062dc9b1a" is_pre_election: true
I20251026 15:12:15.699311 28923 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3549ca467d34532abc6ecc29a893511 in term 0.
I20251026 15:12:15.699625 28821 leader_election.cc:304] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3549ca467d34532abc6ecc29a893511, e650e0beb1b44540aec2430062dc9b1a; no voters:
I20251026 15:12:15.699823 28979 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:12:15.699882 28979 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:12:15.699913 28979 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:15.701241 28979 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } }
I20251026 15:12:15.701391 28979 leader_election.cc:290] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [CANDIDATE]: Term 1 election: Requested vote from peers 2092e6eb477547099dbcb8bf9efd0adb (127.14.212.126:43941), e650e0beb1b44540aec2430062dc9b1a (127.14.212.124:44573)
I20251026 15:12:15.701745 28923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3549ca467d34532abc6ecc29a893511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e650e0beb1b44540aec2430062dc9b1a"
I20251026 15:12:15.701834 28923 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:15.702710 28923 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3549ca467d34532abc6ecc29a893511 in term 1.
I20251026 15:12:15.703040 28768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3549ca467d34532abc6ecc29a893511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2092e6eb477547099dbcb8bf9efd0adb"
I20251026 15:12:15.703182 28768 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:15.703174 28821 leader_election.cc:304] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3549ca467d34532abc6ecc29a893511, e650e0beb1b44540aec2430062dc9b1a; no voters:
I20251026 15:12:15.704139 28768 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3549ca467d34532abc6ecc29a893511 in term 1.
I20251026 15:12:15.703040 28769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3549ca467d34532abc6ecc29a893511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2092e6eb477547099dbcb8bf9efd0adb" is_pre_election: true
I20251026 15:12:15.704301 28769 raft_consensus.cc:2376] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c3549ca467d34532abc6ecc29a893511 in term 1. Re-sending same reply.
I20251026 15:12:15.705641 28979 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:12:15.705862 28979 raft_consensus.cc:697] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [term 1 LEADER]: Becoming Leader. State: Replica: c3549ca467d34532abc6ecc29a893511, State: Running, Role: LEADER
I20251026 15:12:15.705983 28979 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } }
I20251026 15:12:15.706540 28979 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3549ca467d34532abc6ecc29a893511. Latest consensus state: current_term: 1 leader_uuid: "c3549ca467d34532abc6ecc29a893511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } } }
I20251026 15:12:15.706622 28979 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:15.707329 29008 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:12:15.707520 29008 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:12:15.708956 28923 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [term 1 FOLLOWER]: Refusing update from remote peer c3549ca467d34532abc6ecc29a893511: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:12:15.709263 28979 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:12:15.710448 28956 sys_catalog.cc:455] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3549ca467d34532abc6ecc29a893511. Latest consensus state: current_term: 1 leader_uuid: "c3549ca467d34532abc6ecc29a893511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } } }
I20251026 15:12:15.710537 28956 sys_catalog.cc:458] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:12:15.710991 28769 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [term 1 FOLLOWER]: Refusing update from remote peer c3549ca467d34532abc6ecc29a893511: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:12:15.711236 28979 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:12:15.712280 28993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3549ca467d34532abc6ecc29a893511. Latest consensus state: current_term: 1 leader_uuid: "c3549ca467d34532abc6ecc29a893511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } } }
I20251026 15:12:15.712378 28993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:12:15.713604 29006 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3549ca467d34532abc6ecc29a893511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } } }
I20251026 15:12:15.713693 29006 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:15.714182 28993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3549ca467d34532abc6ecc29a893511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } } }
I20251026 15:12:15.714260 28993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:12:15.714483 29009 mvcc.cc:204] Tried to move back new op lower bound from 7215069330261606400 to 7215069330252394496. Current Snapshot: MvccSnapshot[applied={T|T < 7215069330261606400}]
I20251026 15:12:15.714705 29012 mvcc.cc:204] Tried to move back new op lower bound from 7215069330261606400 to 7215069330252394496. Current Snapshot: MvccSnapshot[applied={T|T < 7215069330261606400}]
I20251026 15:12:15.715554 28979 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3549ca467d34532abc6ecc29a893511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } } }
I20251026 15:12:15.715633 28979 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511 [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:15.715986 28956 sys_catalog.cc:455] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3549ca467d34532abc6ecc29a893511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2092e6eb477547099dbcb8bf9efd0adb" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43941 } } peers { permanent_uuid: "c3549ca467d34532abc6ecc29a893511" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 46661 } } peers { permanent_uuid: "e650e0beb1b44540aec2430062dc9b1a" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 44573 } } }
I20251026 15:12:15.716080 28956 sys_catalog.cc:458] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:12:15.717793 29010 mvcc.cc:204] Tried to move back new op lower bound from 7215069330261606400 to 7215069330252394496. Current Snapshot: MvccSnapshot[applied={T|T < 7215069330261606400}]
I20251026 15:12:15.718770 29008 catalog_manager.cc:1357] Generated new cluster ID: 63728d45df0f466bba5430a708f0db6c
I20251026 15:12:15.718823 29008 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:12:15.735502 29008 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:12:15.736361 29008 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:12:15.748343 29008 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c3549ca467d34532abc6ecc29a893511: Generated new TSK 0
I20251026 15:12:15.748593 29008 catalog_manager.cc:1524] Initializing in-progress tserver states...
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.14.212.125@KRBTEST.COM
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.14.212.126@KRBTEST.COM
Oct 26 15:12:15 dist-test-slave-8pgz krb5kdc[28711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.14.212.124@KRBTEST.COM
I20251026 15:12:16.584822 28967 catalog_manager.cc:1269] Loaded cluster ID: 63728d45df0f466bba5430a708f0db6c
I20251026 15:12:16.584910 28967 catalog_manager.cc:1562] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a: loading cluster ID for follower catalog manager: success
I20251026 15:12:16.585810 28967 catalog_manager.cc:1584] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a: acquiring CA information for follower catalog manager: success
I20251026 15:12:16.586127 28967 catalog_manager.cc:1612] T 00000000000000000000000000000000 P e650e0beb1b44540aec2430062dc9b1a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 15:12:16.659614 29004 catalog_manager.cc:1269] Loaded cluster ID: 63728d45df0f466bba5430a708f0db6c
I20251026 15:12:16.659718 29004 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb: loading cluster ID for follower catalog manager: success
I20251026 15:12:16.660661 29004 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb: acquiring CA information for follower catalog manager: success
I20251026 15:12:16.660971 29004 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 2092e6eb477547099dbcb8bf9efd0adb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 15:12:16.870144 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 28727
I20251026 15:12:16.875535 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 28804
I20251026 15:12:16.880966 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 28881
2025-10-26T15:12:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3964 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:12:16 dist-test-slave-8pgz krb5kdc[29023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:12:16 dist-test-slave-8pgz krb5kdc[29023](info): set up 2 sockets
Oct 26 15:12:16 dist-test-slave-8pgz krb5kdc[29023](info): commencing operation
krb5kdc: starting...
W20251026 15:12:18.879881 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.978s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:12:18 dist-test-slave-8pgz krb5kdc[29023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:12:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:19.025717 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:45683
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:45683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:19.108065 29039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:19.108248 29039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:19.108273 29039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:19.109777 29039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:12:19.109828 29039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:19.109843 29039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:19.109858 29039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:12:19.109887 29039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:12:19.111656 29039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:45683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:45683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:19.111898 29039 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:19.112092 29039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:19.115056 29047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:19.115024 29044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:19.115065 29039 server_base.cc:1047] running on GCE node
W20251026 15:12:19.115047 29045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:19.115586 29039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:19.115804 29039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:19.116958 29039 hybrid_clock.cc:648] HybridClock initialized: now 1761491539116945 us; error 29 us; skew 500 ppm
Oct 26 15:12:19 dist-test-slave-8pgz krb5kdc[29023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491539, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:19.119699 29039 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:12:19.120200 29039 webserver.cc:492] Webserver started at http://127.14.212.126:46617/ using document root <none> and password file <none>
I20251026 15:12:19.120401 29039 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:19.120442 29039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:19.120566 29039 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:12:19.121543 29039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "06956de1d9a445d28eca4cbc632a50c1"
format_stamp: "Formatted at 2025-10-26 15:12:19 on dist-test-slave-8pgz"
server_key: "ecc0da99f1d59de1d4013d7c220a2ba4"
server_key_iv: "29c82fe69fd3efbea89e885497a1548c"
server_key_version: "encryptionkey@0"
I20251026 15:12:19.121856 29039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "06956de1d9a445d28eca4cbc632a50c1"
format_stamp: "Formatted at 2025-10-26 15:12:19 on dist-test-slave-8pgz"
server_key: "ecc0da99f1d59de1d4013d7c220a2ba4"
server_key_iv: "29c82fe69fd3efbea89e885497a1548c"
server_key_version: "encryptionkey@0"
I20251026 15:12:19.123558 29039 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20251026 15:12:19.124493 29054 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:19.124711 29039 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:19.124804 29039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "06956de1d9a445d28eca4cbc632a50c1"
format_stamp: "Formatted at 2025-10-26 15:12:19 on dist-test-slave-8pgz"
server_key: "ecc0da99f1d59de1d4013d7c220a2ba4"
server_key_iv: "29c82fe69fd3efbea89e885497a1548c"
server_key_version: "encryptionkey@0"
I20251026 15:12:19.124886 29039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:19.151400 29039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:19.153374 29039 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:19.153483 29039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:19.157508 29039 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:45683
I20251026 15:12:19.157591 29106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:45683 every 8 connection(s)
I20251026 15:12:19.157884 29039 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:12:19.158614 29107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:19.160401 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 29039
I20251026 15:12:19.160499 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:12:19.160676 15185 external_mini_cluster.cc:1468] Setting key c6eaf0b3dbffb7cbfe2b17560820018e
I20251026 15:12:19.161266 29107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1: Bootstrap starting.
I20251026 15:12:19.161969 29107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:19.162231 29107 log.cc:826] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:19.163025 29107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1: No bootstrap required, opened a new log
I20251026 15:12:19.164129 29107 raft_consensus.cc:359] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06956de1d9a445d28eca4cbc632a50c1" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45683 } }
I20251026 15:12:19.164254 29107 raft_consensus.cc:385] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:19.164288 29107 raft_consensus.cc:740] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06956de1d9a445d28eca4cbc632a50c1, State: Initialized, Role: FOLLOWER
I20251026 15:12:19.164439 29107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06956de1d9a445d28eca4cbc632a50c1" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45683 } }
I20251026 15:12:19.164523 29107 raft_consensus.cc:399] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:12:19.164556 29107 raft_consensus.cc:493] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:12:19.164608 29107 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:19.165252 29107 raft_consensus.cc:515] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06956de1d9a445d28eca4cbc632a50c1" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45683 } }
I20251026 15:12:19.165378 29107 leader_election.cc:304] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06956de1d9a445d28eca4cbc632a50c1; no voters:
I20251026 15:12:19.165560 29107 leader_election.cc:290] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:12:19.165617 29112 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:12:19.165721 29112 raft_consensus.cc:697] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [term 1 LEADER]: Becoming Leader. State: Replica: 06956de1d9a445d28eca4cbc632a50c1, State: Running, Role: LEADER
I20251026 15:12:19.165815 29107 sys_catalog.cc:565] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:12:19.165802 29112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06956de1d9a445d28eca4cbc632a50c1" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45683 } }
Oct 26 15:12:19 dist-test-slave-8pgz krb5kdc[29023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:19.166512 29114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06956de1d9a445d28eca4cbc632a50c1. Latest consensus state: current_term: 1 leader_uuid: "06956de1d9a445d28eca4cbc632a50c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06956de1d9a445d28eca4cbc632a50c1" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45683 } } }
I20251026 15:12:19.166704 29114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:19.167183 29113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06956de1d9a445d28eca4cbc632a50c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06956de1d9a445d28eca4cbc632a50c1" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45683 } } }
I20251026 15:12:19.167264 29113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1 [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:19.167351 29121 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:12:19.168354 29121 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:12:19.169983 29121 catalog_manager.cc:1357] Generated new cluster ID: e0fbafa31c074b22b1dfbcfc4542f5b8
I20251026 15:12:19.170030 29121 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:12:19.170732 29110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:19.161166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60492 (local address 127.14.212.126:45683)
1026 15:12:19.161348 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:19.161357 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:19.161382 (+ 25us) server_negotiation.cc:408] Connection header received
1026 15:12:19.161737 (+ 355us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:19.161742 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:19.161784 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:19.161903 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:19.162842 (+ 939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:19.163588 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:19.164168 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:19.164443 (+ 275us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:19.167000 (+ 2557us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:19.167010 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:19.167011 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:19.167036 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:19.169174 (+ 2138us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:19.169584 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:19.169588 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:19.169588 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:19.169662 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:19.169855 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:19.169856 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:19.169857 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:19.170084 (+ 227us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:19.170227 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:19.170368 (+ 141us) server_negotiation.cc:300] Negotiation successful
1026 15:12:19.170532 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20251026 15:12:19.171554 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:12:19.186705 29121 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:12:19.187580 29121 catalog_manager.cc:1514] Loading token signing keys...
Oct 26 15:12:19 dist-test-slave-8pgz krb5kdc[29023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20251026 15:12:19.194700 29121 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 06956de1d9a445d28eca4cbc632a50c1: Generated new TSK 0
I20251026 15:12:19.194906 29121 catalog_manager.cc:1524] Initializing in-progress tserver states...
Oct 26 15:12:19 dist-test-slave-8pgz krb5kdc[29023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.14.212.126@KRBTEST.COM
I20251026 15:12:19.204312 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 29039
2025-10-26T15:12:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2321 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:12:19 dist-test-slave-8pgz krb5kdc[29136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:12:19 dist-test-slave-8pgz krb5kdc[29136](info): set up 2 sockets
Oct 26 15:12:19 dist-test-slave-8pgz krb5kdc[29136](info): commencing operation
krb5kdc: starting...
W20251026 15:12:21.192031 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.969s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:12:21 dist-test-slave-8pgz krb5kdc[29136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:12:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:12:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:21.337349 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:32837
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:32837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:21.418081 29152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:21.418339 29152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:21.418383 29152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:21.420056 29152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:12:21.420127 29152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:21.420150 29152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:21.420173 29152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:12:21.420192 29152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:12:21.422072 29152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:32837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:32837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:21.422370 29152 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:21.422605 29152 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:21.425380 29157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:21.425523 29160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:21.425513 29152 server_base.cc:1047] running on GCE node
W20251026 15:12:21.425422 29158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:21.426755 29152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:21.427058 29152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:21.428200 29152 hybrid_clock.cc:648] HybridClock initialized: now 1761491541428180 us; error 43 us; skew 500 ppm
Oct 26 15:12:21 dist-test-slave-8pgz krb5kdc[29136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491541, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:21.430931 29152 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:12:21.431485 29152 webserver.cc:492] Webserver started at http://127.14.212.126:35029/ using document root <none> and password file <none>
I20251026 15:12:21.431681 29152 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:21.431723 29152 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:21.431842 29152 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:12:21.432811 29152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "a1abd0e586c848209053268370498ae8"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "b23caa226e4d39f0c7c6f25459aab7af"
server_key_iv: "49d83dadc66c053e5b7c59897e558d17"
server_key_version: "encryptionkey@0"
I20251026 15:12:21.433265 29152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "a1abd0e586c848209053268370498ae8"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "b23caa226e4d39f0c7c6f25459aab7af"
server_key_iv: "49d83dadc66c053e5b7c59897e558d17"
server_key_version: "encryptionkey@0"
I20251026 15:12:21.434605 29152 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251026 15:12:21.435341 29167 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:21.435528 29152 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251026 15:12:21.435627 29152 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "a1abd0e586c848209053268370498ae8"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "b23caa226e4d39f0c7c6f25459aab7af"
server_key_iv: "49d83dadc66c053e5b7c59897e558d17"
server_key_version: "encryptionkey@0"
I20251026 15:12:21.435698 29152 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:21.451414 29152 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:21.458074 29152 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:21.458274 29152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:21.462723 29152 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:32837
I20251026 15:12:21.462772 29219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:32837 every 8 connection(s)
I20251026 15:12:21.463106 29152 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:12:21.463846 29220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:21.466490 29220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Bootstrap starting.
I20251026 15:12:21.467109 29220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:21.467367 29220 log.cc:826] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:21.468151 29220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: No bootstrap required, opened a new log
I20251026 15:12:21.469430 29220 raft_consensus.cc:359] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } }
I20251026 15:12:21.469558 29220 raft_consensus.cc:385] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:21.469583 29220 raft_consensus.cc:740] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1abd0e586c848209053268370498ae8, State: Initialized, Role: FOLLOWER
I20251026 15:12:21.469705 29220 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } }
I20251026 15:12:21.469763 29220 raft_consensus.cc:399] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:12:21.469787 29220 raft_consensus.cc:493] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:12:21.469815 29220 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:21.470436 29220 raft_consensus.cc:515] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } }
I20251026 15:12:21.470544 29220 leader_election.cc:304] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1abd0e586c848209053268370498ae8; no voters:
I20251026 15:12:21.470731 29220 leader_election.cc:290] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:12:21.470788 29223 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:12:21.470935 29223 raft_consensus.cc:697] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 1 LEADER]: Becoming Leader. State: Replica: a1abd0e586c848209053268370498ae8, State: Running, Role: LEADER
I20251026 15:12:21.470971 29220 sys_catalog.cc:565] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:12:21.471022 29223 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } }
I20251026 15:12:21.471351 29225 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1abd0e586c848209053268370498ae8. Latest consensus state: current_term: 1 leader_uuid: "a1abd0e586c848209053268370498ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } } }
I20251026 15:12:21.471470 29225 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:21.471364 29224 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1abd0e586c848209053268370498ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } } }
I20251026 15:12:21.471602 29224 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:21.471688 29229 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:12:21.472210 29229 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:12:21.472363 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 29152
I20251026 15:12:21.472465 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:12:21.472692 15185 external_mini_cluster.cc:1468] Setting key 98168008446713daedecd87e73809d85
I20251026 15:12:21.474167 29229 catalog_manager.cc:1357] Generated new cluster ID: 858aaa487627451483b66409cdc05e25
I20251026 15:12:21.474221 29229 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 26 15:12:21 dist-test-slave-8pgz krb5kdc[29136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:21.481727 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:21.473201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38194 (local address 127.14.212.126:32837)
1026 15:12:21.473372 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:21.473383 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:21.473409 (+ 26us) server_negotiation.cc:408] Connection header received
1026 15:12:21.473581 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:21.473587 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:21.473641 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:21.473777 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:21.474811 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:21.475595 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:21.476161 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:21.476768 (+ 607us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:21.478624 (+ 1856us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:21.478630 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:21.478630 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:21.478649 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:21.480381 (+ 1732us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:21.480711 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:21.480715 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:21.480715 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:21.480778 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:21.480930 (+ 152us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:21.480931 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:21.480931 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:21.481142 (+ 211us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:21.481288 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:21.481390 (+ 102us) server_negotiation.cc:300] Negotiation successful
1026 15:12:21.481525 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20251026 15:12:21.486114 29229 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:12:21.486912 29229 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:12:21.491967 29229 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Generated new TSK 0
I20251026 15:12:21.492177 29229 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:21.541321 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:32837
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:21.623926 29248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:21.624199 29248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:21.624243 29248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:21.625725 29248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:21.625771 29248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:21.625818 29248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:12:21.627542 29248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:32837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:21.627790 29248 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:21.627975 29248 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:21.630683 29256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:21.630664 29254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:21.630653 29253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:21.630963 29248 server_base.cc:1047] running on GCE node
I20251026 15:12:21.631126 29248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:21.631338 29248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:21.632490 29248 hybrid_clock.cc:648] HybridClock initialized: now 1761491541632476 us; error 32 us; skew 500 ppm
Oct 26 15:12:21 dist-test-slave-8pgz krb5kdc[29136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491541, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:21.634925 29248 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:12:21.635483 29248 webserver.cc:492] Webserver started at http://127.14.212.65:38967/ using document root <none> and password file <none>
I20251026 15:12:21.635689 29248 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:21.635725 29248 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:21.635836 29248 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:12:21.636778 29248 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "3717f5812abb4c8cae8c7f1ca081ee69"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "773c601391b75ed3869536b2ad2bcfba"
server_key_iv: "36b4acbf513c15ea5061d1d4a3d46bf8"
server_key_version: "encryptionkey@0"
I20251026 15:12:21.637069 29248 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "3717f5812abb4c8cae8c7f1ca081ee69"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "773c601391b75ed3869536b2ad2bcfba"
server_key_iv: "36b4acbf513c15ea5061d1d4a3d46bf8"
server_key_version: "encryptionkey@0"
I20251026 15:12:21.638273 29248 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20251026 15:12:21.639063 29263 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:21.639251 29248 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:12:21.639372 29248 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "3717f5812abb4c8cae8c7f1ca081ee69"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "773c601391b75ed3869536b2ad2bcfba"
server_key_iv: "36b4acbf513c15ea5061d1d4a3d46bf8"
server_key_version: "encryptionkey@0"
I20251026 15:12:21.639451 29248 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:21.657737 29248 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:21.660118 29248 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:21.660239 29248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:21.660435 29248 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:21.660771 29248 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:21.660807 29248 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:21.660843 29248 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:21.660862 29248 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:21.666880 29248 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:44075
I20251026 15:12:21.666934 29376 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:44075 every 8 connection(s)
I20251026 15:12:21.667244 29248 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
Oct 26 15:12:21 dist-test-slave-8pgz krb5kdc[29136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491541, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:21.675930 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:21.667959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:40727 (local address 127.14.212.126:32837)
1026 15:12:21.668027 (+ 68us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:21.668030 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:21.668211 (+ 181us) server_negotiation.cc:408] Connection header received
1026 15:12:21.668756 (+ 545us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:21.668757 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:21.668775 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:21.668835 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:21.669644 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:21.670133 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:21.670739 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:21.670855 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:21.673268 (+ 2413us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:21.673274 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:21.673274 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:21.673281 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:21.674644 (+ 1363us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:21.675032 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:21.675033 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:21.675033 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:21.675051 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:21.675286 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:21.675287 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:21.675287 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:21.675392 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:21.675448 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:21.675746 (+ 298us) server_negotiation.cc:300] Negotiation successful
1026 15:12:21.675860 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251026 15:12:21.675992 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:21.667920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:40727)
1026 15:12:21.668072 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:21.668103 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:21.668666 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:21.668894 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:21.668898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:21.669224 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:21.669551 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:21.669555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:21.670216 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:21.670217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:21.670665 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:21.670667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:21.670777 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:21.671759 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:21.671777 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:21.673161 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:21.674728 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:21.674735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:21.674738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:21.674959 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:21.675131 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:21.675132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:21.675133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:21.675217 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:21.675445 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:21.675448 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:21.675583 (+ 135us) client_negotiation.cc:769] Sending connection context
1026 15:12:21.675679 (+ 96us) client_negotiation.cc:240] Negotiation successful
1026 15:12:21.675809 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20251026 15:12:21.676559 29377 heartbeater.cc:344] Connected to a master server at 127.14.212.126:32837
I20251026 15:12:21.676692 29377 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:21.676851 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 29248
I20251026 15:12:21.676957 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:12:21.676972 29377 heartbeater.cc:507] Master 127.14.212.126:32837 requested a full tablet report, sending...
I20251026 15:12:21.677183 15185 external_mini_cluster.cc:1468] Setting key 5d164a39bb9d74f9acbf1c988701e590
I20251026 15:12:21.677560 29184 ts_manager.cc:194] Registered new tserver with Master: 3717f5812abb4c8cae8c7f1ca081ee69 (127.14.212.65:44075)
I20251026 15:12:21.678274 29184 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:40727
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:21.730161 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:32837
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:21.810811 29384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:21.811041 29384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:21.811082 29384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:21.812616 29384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:21.812680 29384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:21.812724 29384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:12:21.814560 29384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:32837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:21.814851 29384 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:21.815063 29384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:21.817924 29392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:21.818034 29390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:21.817965 29389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:21.818159 29384 server_base.cc:1047] running on GCE node
I20251026 15:12:21.818599 29384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:21.818849 29384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:21.820041 29384 hybrid_clock.cc:648] HybridClock initialized: now 1761491541820025 us; error 38 us; skew 500 ppm
Oct 26 15:12:21 dist-test-slave-8pgz krb5kdc[29136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491541, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:21.823009 29384 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:12:21.823635 29384 webserver.cc:492] Webserver started at http://127.14.212.66:33563/ using document root <none> and password file <none>
I20251026 15:12:21.823825 29384 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:21.823863 29384 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:21.823963 29384 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:12:21.825089 29384 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "54d141fdf2874e50b643d9dafaf62410"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "2d9e6e0cd91423a4229285946303524f"
server_key_iv: "b3ef628c0d497d8d4a34ba1f17d9521c"
server_key_version: "encryptionkey@0"
I20251026 15:12:21.825413 29384 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "54d141fdf2874e50b643d9dafaf62410"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "2d9e6e0cd91423a4229285946303524f"
server_key_iv: "b3ef628c0d497d8d4a34ba1f17d9521c"
server_key_version: "encryptionkey@0"
I20251026 15:12:21.826718 29384 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:12:21.827490 29399 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:21.827672 29384 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:12:21.827741 29384 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "54d141fdf2874e50b643d9dafaf62410"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "2d9e6e0cd91423a4229285946303524f"
server_key_iv: "b3ef628c0d497d8d4a34ba1f17d9521c"
server_key_version: "encryptionkey@0"
I20251026 15:12:21.827811 29384 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:21.848683 29384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:21.850847 29384 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:21.850978 29384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:21.851223 29384 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:21.851544 29384 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:21.851584 29384 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:21.851609 29384 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:21.851634 29384 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:21.857434 29384 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:35933
I20251026 15:12:21.857496 29512 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:35933 every 8 connection(s)
I20251026 15:12:21.857823 29384 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
Oct 26 15:12:21 dist-test-slave-8pgz krb5kdc[29136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491541, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:21.865099 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 29384
I20251026 15:12:21.865234 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:12:21.865449 15185 external_mini_cluster.cc:1468] Setting key 07b44426f33e098e08b8afbe49297865
I20251026 15:12:21.866716 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:21.858617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:39109 (local address 127.14.212.126:32837)
1026 15:12:21.858688 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:21.858690 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:21.858857 (+ 167us) server_negotiation.cc:408] Connection header received
1026 15:12:21.859342 (+ 485us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:21.859343 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:21.859354 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:21.859415 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:21.860203 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:21.860702 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:21.861244 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:21.861346 (+ 102us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:21.863787 (+ 2441us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:21.863792 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:21.863792 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:21.863800 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:21.865255 (+ 1455us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:21.865790 (+ 535us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:21.865791 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:21.865791 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:21.865826 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:21.865986 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:21.865987 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:21.865987 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:21.866100 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:21.866176 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:21.866541 (+ 365us) server_negotiation.cc:300] Negotiation successful
1026 15:12:21.866651 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251026 15:12:21.866962 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:21.858571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:39109)
1026 15:12:21.858740 (+ 169us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:21.858769 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:21.859265 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:21.859486 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:21.859493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:21.859837 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:21.860127 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:21.860130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:21.860794 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:21.860795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:21.861179 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:21.861181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:21.861297 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:21.862259 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:21.862276 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:21.863692 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:21.865364 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:21.865372 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:21.865376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:21.865695 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:21.865876 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:21.865877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:21.865878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:21.865938 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:21.866166 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:21.866170 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:21.866351 (+ 181us) client_negotiation.cc:769] Sending connection context
1026 15:12:21.866508 (+ 157us) client_negotiation.cc:240] Negotiation successful
1026 15:12:21.866681 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20251026 15:12:21.867583 29513 heartbeater.cc:344] Connected to a master server at 127.14.212.126:32837
I20251026 15:12:21.867712 29513 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:21.867928 29513 heartbeater.cc:507] Master 127.14.212.126:32837 requested a full tablet report, sending...
I20251026 15:12:21.868330 29184 ts_manager.cc:194] Registered new tserver with Master: 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933)
I20251026 15:12:21.868741 29184 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:39109
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:21.913813 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:32837
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:21.997970 29520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:21.998194 29520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:21.998222 29520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:21.999774 29520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:21.999833 29520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:21.999863 29520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:12:22.001523 29520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:32837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:22.001761 29520 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:22.001927 29520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:22.004812 29526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:22.004875 29525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:22.004968 29520 server_base.cc:1047] running on GCE node
W20251026 15:12:22.004813 29528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:22.005208 29520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:22.005429 29520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:22.006583 29520 hybrid_clock.cc:648] HybridClock initialized: now 1761491542006560 us; error 32 us; skew 500 ppm
Oct 26 15:12:22 dist-test-slave-8pgz krb5kdc[29136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491542, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:22.009196 29520 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:12:22.009721 29520 webserver.cc:492] Webserver started at http://127.14.212.67:35515/ using document root <none> and password file <none>
I20251026 15:12:22.009886 29520 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:22.009918 29520 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:22.009999 29520 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:12:22.010980 29520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "d2be86aaf16042299571a85097485a13"
format_stamp: "Formatted at 2025-10-26 15:12:22 on dist-test-slave-8pgz"
server_key: "d305f831ccbc6c3c47800ec2534d43e2"
server_key_iv: "d51cdb92175fa0a39440c66eea9ee634"
server_key_version: "encryptionkey@0"
I20251026 15:12:22.011260 29520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "d2be86aaf16042299571a85097485a13"
format_stamp: "Formatted at 2025-10-26 15:12:22 on dist-test-slave-8pgz"
server_key: "d305f831ccbc6c3c47800ec2534d43e2"
server_key_iv: "d51cdb92175fa0a39440c66eea9ee634"
server_key_version: "encryptionkey@0"
I20251026 15:12:22.012504 29520 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:12:22.013234 29535 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.013494 29520 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:22.013581 29520 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "d2be86aaf16042299571a85097485a13"
format_stamp: "Formatted at 2025-10-26 15:12:22 on dist-test-slave-8pgz"
server_key: "d305f831ccbc6c3c47800ec2534d43e2"
server_key_iv: "d51cdb92175fa0a39440c66eea9ee634"
server_key_version: "encryptionkey@0"
I20251026 15:12:22.013639 29520 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:22.026857 29520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:22.028959 29520 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:22.029090 29520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:22.029325 29520 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:22.029670 29520 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:22.029703 29520 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.029726 29520 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:22.029739 29520 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.035864 29520 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:37101
I20251026 15:12:22.035998 29648 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:37101 every 8 connection(s)
I20251026 15:12:22.036233 29520 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:12:22.038625 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 29520
I20251026 15:12:22.038712 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:12:22.038913 15185 external_mini_cluster.cc:1468] Setting key f92fd21be69646166daa24e8796769c8
Oct 26 15:12:22 dist-test-slave-8pgz krb5kdc[29136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491542, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:22.045126 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.036919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:38575 (local address 127.14.212.126:32837)
1026 15:12:22.036996 (+ 77us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.036999 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.037120 (+ 121us) server_negotiation.cc:408] Connection header received
1026 15:12:22.037641 (+ 521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.037642 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.037654 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.037710 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:22.038519 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.039091 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.039750 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.039915 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.042473 (+ 2558us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:22.042480 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:22.042480 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:22.042488 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:22.044066 (+ 1578us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:22.044481 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:22.044482 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:22.044482 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:22.044499 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:22.044642 (+ 143us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:22.044642 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:22.044642 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:22.044729 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:22.044774 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.044987 (+ 213us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.045052 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251026 15:12:22.045269 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.036873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:38575)
1026 15:12:22.037016 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.037041 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.037537 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.037755 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.037758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.038075 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:22.038382 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.038385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.039181 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.039182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.039679 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.039681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.039824 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.040823 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:22.040840 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:22.042279 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:22.044150 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:22.044157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:22.044159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:22.044417 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:22.044544 (+ 127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:22.044544 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:22.044545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:22.044600 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:22.044769 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:22.044771 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:22.044889 (+ 118us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.044972 (+ 83us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.045079 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20251026 15:12:22.045749 29649 heartbeater.cc:344] Connected to a master server at 127.14.212.126:32837
I20251026 15:12:22.045853 29649 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:22.046051 29649 heartbeater.cc:507] Master 127.14.212.126:32837 requested a full tablet report, sending...
I20251026 15:12:22.046502 29184 ts_manager.cc:194] Registered new tserver with Master: d2be86aaf16042299571a85097485a13 (127.14.212.67:37101)
I20251026 15:12:22.046885 29184 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:38575
I20251026 15:12:22.050801 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:12:22.057829 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.052120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38206 (local address 127.14.212.126:32837)
1026 15:12:22.052190 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.052191 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.052245 (+ 54us) server_negotiation.cc:408] Connection header received
1026 15:12:22.052377 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.052378 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.052387 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.052436 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:22.053119 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.053611 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.054146 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.054246 (+ 100us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.055391 (+ 1145us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:22.055396 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:22.055397 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:22.055403 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:22.056821 (+ 1418us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:22.057150 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:22.057151 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:22.057151 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:22.057169 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:22.057275 (+ 106us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:22.057276 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:22.057276 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:22.057439 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:22.057520 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.057680 (+ 160us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.057753 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251026 15:12:22.059811 29184 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:12:22.061164 29184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:12:22.066799 29664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.063709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:37101 (local address 127.0.0.1:44794)
1026 15:12:22.064292 (+ 583us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.064303 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.064364 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.064689 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.064689 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.064691 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:22.064840 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.064841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.065703 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.065704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.066455 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.066457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.066518 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.066574 (+ 56us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.066631 (+ 57us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.066672 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":55,"threads_started":1}
I20251026 15:12:22.066931 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.063537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:35933 (local address 127.0.0.1:49828)
1026 15:12:22.063805 (+ 268us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.063814 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.063858 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.064388 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.064389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.064391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:22.064607 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.064609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.065869 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.065869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.066745 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.066748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.066808 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.066811 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.066841 (+ 30us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.066876 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":80,"threads_started":1}
I20251026 15:12:22.067476 29665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.063821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.14.212.67:37101)
1026 15:12:22.064014 (+ 193us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.064021 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.064485 (+ 464us) server_negotiation.cc:408] Connection header received
1026 15:12:22.064528 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.064531 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.064548 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.064609 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:22.064918 (+ 309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.065620 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.066711 (+ 1091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.067321 (+ 610us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.067366 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.067392 (+ 26us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.067420 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
I20251026 15:12:22.067797 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.063449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.14.212.65:44075)
1026 15:12:22.063629 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.063638 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.063667 (+ 29us) server_negotiation.cc:408] Connection header received
1026 15:12:22.063706 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.063708 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.063726 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.063787 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:22.064323 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.065496 (+ 1173us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.066778 (+ 1282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.067584 (+ 806us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.067640 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.067671 (+ 31us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.067718 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20251026 15:12:22.067986 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.063360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:44075 (local address 127.0.0.1:46850)
1026 15:12:22.063477 (+ 117us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.063508 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.063623 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.063896 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.063899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.063901 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:22.064214 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.064216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.065583 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.065584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.066506 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.066509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.066579 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.066584 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.066624 (+ 40us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.066666 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":63,"threads_started":1}
I20251026 15:12:22.068095 29583 tablet_service.cc:1505] Processing CreateTablet for tablet 07f44eb3a34845df84aaec042cbc0327 (DEFAULT_TABLE table=test-table [id=a1d673fd89c14a7db09ef29df2d760be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:12:22.068202 29663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.063655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.14.212.66:35933)
1026 15:12:22.064045 (+ 390us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.064053 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.064064 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:12:22.064101 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.064103 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.064118 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.064190 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:22.064705 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.065793 (+ 1088us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.067123 (+ 1330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.067896 (+ 773us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.067947 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.067976 (+ 29us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.068025 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":80,"threads_started":1}
I20251026 15:12:22.068415 29583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f44eb3a34845df84aaec042cbc0327. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:22.068555 29311 tablet_service.cc:1505] Processing CreateTablet for tablet 07f44eb3a34845df84aaec042cbc0327 (DEFAULT_TABLE table=test-table [id=a1d673fd89c14a7db09ef29df2d760be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:12:22.068637 29447 tablet_service.cc:1505] Processing CreateTablet for tablet 07f44eb3a34845df84aaec042cbc0327 (DEFAULT_TABLE table=test-table [id=a1d673fd89c14a7db09ef29df2d760be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:12:22.068843 29447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f44eb3a34845df84aaec042cbc0327. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:22.068843 29311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f44eb3a34845df84aaec042cbc0327. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:22.071244 29666 tablet_bootstrap.cc:492] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13: Bootstrap starting.
I20251026 15:12:22.071605 29668 tablet_bootstrap.cc:492] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410: Bootstrap starting.
I20251026 15:12:22.072273 29668 tablet_bootstrap.cc:654] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:22.072273 29666 tablet_bootstrap.cc:654] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:22.072602 29666 log.cc:826] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:22.072613 29668 log.cc:826] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:22.072923 29667 tablet_bootstrap.cc:492] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Bootstrap starting.
I20251026 15:12:22.073406 29668 tablet_bootstrap.cc:492] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410: No bootstrap required, opened a new log
I20251026 15:12:22.073482 29667 tablet_bootstrap.cc:654] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:22.073493 29666 tablet_bootstrap.cc:492] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13: No bootstrap required, opened a new log
I20251026 15:12:22.073527 29668 ts_tablet_manager.cc:1403] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:12:22.073568 29666 ts_tablet_manager.cc:1403] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:12:22.073804 29667 log.cc:826] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:22.074599 29667 tablet_bootstrap.cc:492] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: No bootstrap required, opened a new log
I20251026 15:12:22.074669 29667 ts_tablet_manager.cc:1403] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:12:22.074970 29666 raft_consensus.cc:359] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } } peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } }
I20251026 15:12:22.075094 29666 raft_consensus.cc:385] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:22.075125 29666 raft_consensus.cc:740] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2be86aaf16042299571a85097485a13, State: Initialized, Role: FOLLOWER
I20251026 15:12:22.075297 29666 consensus_queue.cc:260] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } } peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } }
I20251026 15:12:22.075345 29668 raft_consensus.cc:359] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } } peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } }
I20251026 15:12:22.075472 29668 raft_consensus.cc:385] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:22.075500 29668 raft_consensus.cc:740] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d141fdf2874e50b643d9dafaf62410, State: Initialized, Role: FOLLOWER
I20251026 15:12:22.075605 29668 consensus_queue.cc:260] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } } peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } }
I20251026 15:12:22.075740 29666 ts_tablet_manager.cc:1434] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:12:22.075848 29668 ts_tablet_manager.cc:1434] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:12:22.075860 29649 heartbeater.cc:499] Master 127.14.212.126:32837 was elected leader, sending a full tablet report...
I20251026 15:12:22.075982 29513 heartbeater.cc:499] Master 127.14.212.126:32837 was elected leader, sending a full tablet report...
I20251026 15:12:22.076565 29667 raft_consensus.cc:359] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } } peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } }
I20251026 15:12:22.076711 29667 raft_consensus.cc:385] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:22.076756 29667 raft_consensus.cc:740] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3717f5812abb4c8cae8c7f1ca081ee69, State: Initialized, Role: FOLLOWER
I20251026 15:12:22.076741 29673 raft_consensus.cc:493] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:12:22.076804 29673 raft_consensus.cc:515] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } } peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } }
I20251026 15:12:22.076855 29667 consensus_queue.cc:260] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } } peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } }
I20251026 15:12:22.077086 29667 ts_tablet_manager.cc:1434] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251026 15:12:22.077118 29377 heartbeater.cc:499] Master 127.14.212.126:32837 was elected leader, sending a full tablet report...
I20251026 15:12:22.077826 29673 leader_election.cc:290] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3717f5812abb4c8cae8c7f1ca081ee69 (127.14.212.65:44075), d2be86aaf16042299571a85097485a13 (127.14.212.67:37101)
I20251026 15:12:22.080016 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.077236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:44075 (local address 127.14.212.66:33437)
1026 15:12:22.077296 (+ 60us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.077308 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.077353 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.077653 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.077654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.077657 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:22.077886 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.077889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.078962 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.078962 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.079795 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.079797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.079874 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.079878 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.079910 (+ 32us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.079948 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20251026 15:12:22.080642 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.078036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:37101 (local address 127.14.212.66:52903)
1026 15:12:22.078223 (+ 187us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.078233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.078302 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.078487 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.078488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.078490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:22.078787 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.078789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.079423 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.079424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.080407 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.080410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.080477 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.080480 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.080511 (+ 31us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.080566 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":114,"threads_started":1}
I20251026 15:12:22.080801 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.077412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:33437 (local address 127.14.212.65:44075)
1026 15:12:22.077497 (+ 85us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.077499 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.077508 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:12:22.077532 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.077555 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.077564 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.077609 (+ 45us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:22.077996 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.078893 (+ 897us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.080136 (+ 1243us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.080652 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.080662 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.080686 (+ 24us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.080725 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251026 15:12:22.081068 29665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.078117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:52903 (local address 127.14.212.67:37101)
1026 15:12:22.078173 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.078175 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.078306 (+ 131us) server_negotiation.cc:408] Connection header received
1026 15:12:22.078347 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.078348 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.078358 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.078442 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:22.078852 (+ 410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.079303 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.080509 (+ 1206us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.080974 (+ 465us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.080981 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.081001 (+ 20us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.081024 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251026 15:12:22.081084 29331 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07f44eb3a34845df84aaec042cbc0327" candidate_uuid: "54d141fdf2874e50b643d9dafaf62410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" is_pre_election: true
I20251026 15:12:22.081223 29331 raft_consensus.cc:2468] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54d141fdf2874e50b643d9dafaf62410 in term 0.
I20251026 15:12:22.081400 29603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07f44eb3a34845df84aaec042cbc0327" candidate_uuid: "54d141fdf2874e50b643d9dafaf62410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2be86aaf16042299571a85097485a13" is_pre_election: true
I20251026 15:12:22.081450 29400 leader_election.cc:304] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3717f5812abb4c8cae8c7f1ca081ee69, 54d141fdf2874e50b643d9dafaf62410; no voters:
I20251026 15:12:22.081534 29603 raft_consensus.cc:2468] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54d141fdf2874e50b643d9dafaf62410 in term 0.
I20251026 15:12:22.081571 29673 raft_consensus.cc:2804] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:12:22.081651 29673 raft_consensus.cc:493] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:12:22.081683 29673 raft_consensus.cc:3060] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:22.082325 29673 raft_consensus.cc:515] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } } peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } }
I20251026 15:12:22.082469 29673 leader_election.cc:290] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [CANDIDATE]: Term 1 election: Requested vote from peers 3717f5812abb4c8cae8c7f1ca081ee69 (127.14.212.65:44075), d2be86aaf16042299571a85097485a13 (127.14.212.67:37101)
I20251026 15:12:22.082700 29331 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07f44eb3a34845df84aaec042cbc0327" candidate_uuid: "54d141fdf2874e50b643d9dafaf62410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3717f5812abb4c8cae8c7f1ca081ee69"
I20251026 15:12:22.082782 29331 raft_consensus.cc:3060] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:22.082753 29603 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07f44eb3a34845df84aaec042cbc0327" candidate_uuid: "54d141fdf2874e50b643d9dafaf62410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2be86aaf16042299571a85097485a13"
I20251026 15:12:22.082832 29603 raft_consensus.cc:3060] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:22.083451 29331 raft_consensus.cc:2468] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54d141fdf2874e50b643d9dafaf62410 in term 1.
I20251026 15:12:22.083568 29603 raft_consensus.cc:2468] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54d141fdf2874e50b643d9dafaf62410 in term 1.
I20251026 15:12:22.083624 29400 leader_election.cc:304] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3717f5812abb4c8cae8c7f1ca081ee69, 54d141fdf2874e50b643d9dafaf62410; no voters:
I20251026 15:12:22.083720 29673 raft_consensus.cc:2804] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:12:22.083859 29673 raft_consensus.cc:697] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [term 1 LEADER]: Becoming Leader. State: Replica: 54d141fdf2874e50b643d9dafaf62410, State: Running, Role: LEADER
I20251026 15:12:22.083958 29673 consensus_queue.cc:237] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } } peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } }
I20251026 15:12:22.084733 29184 catalog_manager.cc:5649] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66). New cstate: current_term: 1 leader_uuid: "54d141fdf2874e50b643d9dafaf62410" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } health_report { overall_health: UNKNOWN } } }
I20251026 15:12:22.095238 29663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.093178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.14.212.66:35933)
1026 15:12:22.093249 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.093251 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.093260 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:12:22.093303 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.093304 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.093321 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.093373 (+ 52us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:12:22.093646 (+ 273us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.094117 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.094716 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.094867 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.094906 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:12:22.095000 (+ 94us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:12:22.095077 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.095135 (+ 58us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.095159 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251026 15:12:22.096915 29331 raft_consensus.cc:1275] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Refusing update from remote peer 54d141fdf2874e50b643d9dafaf62410: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:12:22.096927 29603 raft_consensus.cc:1275] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13 [term 1 FOLLOWER]: Refusing update from remote peer 54d141fdf2874e50b643d9dafaf62410: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:12:22.097265 29676 consensus_queue.cc:1048] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:12:22.097407 29673 consensus_queue.cc:1048] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:12:22.099881 29679 mvcc.cc:204] Tried to move back new op lower bound from 7215069356426457088 to 7215069356376510464. Current Snapshot: MvccSnapshot[applied={T|T < 7215069356426457088}]
I20251026 15:12:22.100566 29680 mvcc.cc:204] Tried to move back new op lower bound from 7215069356426457088 to 7215069356376510464. Current Snapshot: MvccSnapshot[applied={T|T < 7215069356426457088}]
I20251026 15:12:22.102497 29183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38206:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:12:22.102593 29183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38206:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:12:22.103395 29183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Sending DeleteTablet for 3 replicas of tablet 07f44eb3a34845df84aaec042cbc0327
I20251026 15:12:22.103737 29311 tablet_service.cc:1552] Processing DeleteTablet for tablet 07f44eb3a34845df84aaec042cbc0327 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:22 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:46850
I20251026 15:12:22.103749 29447 tablet_service.cc:1552] Processing DeleteTablet for tablet 07f44eb3a34845df84aaec042cbc0327 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:22 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:49828
I20251026 15:12:22.103830 29583 tablet_service.cc:1552] Processing DeleteTablet for tablet 07f44eb3a34845df84aaec042cbc0327 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:22 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:44794
I20251026 15:12:22.104009 29689 tablet_replica.cc:333] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410: stopping tablet replica
I20251026 15:12:22.104069 29691 tablet_replica.cc:333] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13: stopping tablet replica
I20251026 15:12:22.104094 29689 raft_consensus.cc:2243] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:12:22.104101 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 29248
I20251026 15:12:22.104152 29691 raft_consensus.cc:2243] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:12:22.104205 29689 raft_consensus.cc:2272] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:12:22.104245 29691 raft_consensus.cc:2272] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:12:22.104784 29691 ts_tablet_manager.cc:1916] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:12:22.106809 29691 ts_tablet_manager.cc:1929] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:12:22.106887 29691 log.cc:1199] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/07f44eb3a34845df84aaec042cbc0327
I20251026 15:12:22.107169 29691 ts_tablet_manager.cc:1950] T 07f44eb3a34845df84aaec042cbc0327 P d2be86aaf16042299571a85097485a13: Deleting consensus metadata
I20251026 15:12:22.107600 29168 catalog_manager.cc:4985] TS d2be86aaf16042299571a85097485a13 (127.14.212.67:37101): tablet 07f44eb3a34845df84aaec042cbc0327 (table test-table [id=a1d673fd89c14a7db09ef29df2d760be]) successfully deleted
I20251026 15:12:22.107908 29689 ts_tablet_manager.cc:1916] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:12:22.109881 29689 ts_tablet_manager.cc:1929] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:12:22.109956 29689 log.cc:1199] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/07f44eb3a34845df84aaec042cbc0327
I20251026 15:12:22.110230 29689 ts_tablet_manager.cc:1950] T 07f44eb3a34845df84aaec042cbc0327 P 54d141fdf2874e50b643d9dafaf62410: Deleting consensus metadata
I20251026 15:12:22.110704 29168 catalog_manager.cc:4985] TS 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933): tablet 07f44eb3a34845df84aaec042cbc0327 (table test-table [id=a1d673fd89c14a7db09ef29df2d760be]) successfully deleted
W20251026 15:12:22.111019 29168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:44075 (error 108)
I20251026 15:12:22.111264 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 29384
I20251026 15:12:22.111892 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.111438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:44075 (local address 127.0.0.1:46862)
1026 15:12:22.111819 (+ 381us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.111853 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:44075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361}
W20251026 15:12:22.112082 29168 catalog_manager.cc:4712] TS 3717f5812abb4c8cae8c7f1ca081ee69 (127.14.212.65:44075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07f44eb3a34845df84aaec042cbc0327: Network error: Client connection negotiation failed: client connection to 127.14.212.65:44075: connect: Connection refused (error 111)
I20251026 15:12:22.115999 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 29520
I20251026 15:12:22.120551 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 29152
I20251026 15:12:22.126240 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:32837
--webserver_interface=127.14.212.126
--webserver_port=35029
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:32837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:22.206832 29693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:22.207036 29693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:22.207074 29693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:22.208518 29693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:12:22.208568 29693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:22.208590 29693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:22.208617 29693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:12:22.208637 29693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:12:22.210282 29693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:32837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:32837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=35029
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:22.210565 29693 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:22.210762 29693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:22.216962 29699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:22.216956 29700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:22.217109 29693 server_base.cc:1047] running on GCE node
W20251026 15:12:22.216955 29702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:22.217460 29693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:22.217742 29693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:22.218912 29693 hybrid_clock.cc:648] HybridClock initialized: now 1761491542218892 us; error 34 us; skew 500 ppm
Oct 26 15:12:22 dist-test-slave-8pgz krb5kdc[29136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491542, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:22.221700 29693 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:12:22.222364 29693 webserver.cc:492] Webserver started at http://127.14.212.126:35029/ using document root <none> and password file <none>
I20251026 15:12:22.222699 29693 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:22.222756 29693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:22.224148 29693 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:12:22.224887 29709 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.225092 29693 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:12:22.225180 29693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "a1abd0e586c848209053268370498ae8"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "b23caa226e4d39f0c7c6f25459aab7af"
server_key_iv: "49d83dadc66c053e5b7c59897e558d17"
server_key_version: "encryptionkey@0"
I20251026 15:12:22.225539 29693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:22.240842 29693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:22.243146 29693 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:22.243346 29693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:22.247417 29693 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:32837
I20251026 15:12:22.247445 29761 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:32837 every 8 connection(s)
I20251026 15:12:22.247802 29693 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:12:22.248975 29762 sys_catalog.cc:263] Verifying existing consensus state
I20251026 15:12:22.249581 29762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Bootstrap starting.
I20251026 15:12:22.250893 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 29693
I20251026 15:12:22.251286 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:44075
--local_ip_for_outbound_sockets=127.14.212.65
--tserver_master_addrs=127.14.212.126:32837
--webserver_port=38967
--webserver_interface=127.14.212.65
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:12:22.260689 29762 log.cc:826] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:22.262722 29762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 15:12:22.263015 29762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Bootstrap complete.
I20251026 15:12:22.265208 29762 raft_consensus.cc:359] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } }
I20251026 15:12:22.265424 29762 raft_consensus.cc:740] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1abd0e586c848209053268370498ae8, State: Initialized, Role: FOLLOWER
I20251026 15:12:22.265563 29762 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } }
I20251026 15:12:22.265640 29762 raft_consensus.cc:399] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:12:22.265692 29762 raft_consensus.cc:493] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:12:22.265748 29762 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 1 FOLLOWER]: Advancing to term 2
I20251026 15:12:22.266738 29762 raft_consensus.cc:515] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } }
I20251026 15:12:22.266848 29762 leader_election.cc:304] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1abd0e586c848209053268370498ae8; no voters:
I20251026 15:12:22.267016 29762 leader_election.cc:290] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251026 15:12:22.267066 29766 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 15:12:22.267234 29766 raft_consensus.cc:697] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 2 LEADER]: Becoming Leader. State: Replica: a1abd0e586c848209053268370498ae8, State: Running, Role: LEADER
I20251026 15:12:22.267277 29762 sys_catalog.cc:565] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:12:22.267403 29766 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } }
I20251026 15:12:22.267823 29767 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a1abd0e586c848209053268370498ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } } }
I20251026 15:12:22.267863 29768 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1abd0e586c848209053268370498ae8. Latest consensus state: current_term: 2 leader_uuid: "a1abd0e586c848209053268370498ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } } }
I20251026 15:12:22.267956 29768 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:22.267915 29767 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:22.268216 29774 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:12:22.269035 29774 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:12:22.269212 29774 catalog_manager.cc:1269] Loaded cluster ID: 858aaa487627451483b66409cdc05e25
I20251026 15:12:22.269443 29774 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:12:22.269666 29774 catalog_manager.cc:1324] IPKI private key was stored in cleartext, attempting to encrypt it.
I20251026 15:12:22.271232 29774 catalog_manager.cc:1334] IPKI private key successfully encrypted.
I20251026 15:12:22.273073 29774 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:12:22.273270 29774 catalog_manager.cc:6033] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Loaded TSK: 0
I20251026 15:12:22.273463 29774 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251026 15:12:22.362205 29764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:22.362470 29764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:22.362510 29764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:22.364758 29764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:22.364823 29764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:22.364868 29764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:12:22.367373 29764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:44075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=38967
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:32837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:22.367631 29764 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:22.367856 29764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:22.370806 29790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:22.370822 29789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:22.370828 29792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:22.371085 29764 server_base.cc:1047] running on GCE node
I20251026 15:12:22.371248 29764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:22.371486 29764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:22.372651 29764 hybrid_clock.cc:648] HybridClock initialized: now 1761491542372637 us; error 32 us; skew 500 ppm
Oct 26 15:12:22 dist-test-slave-8pgz krb5kdc[29136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491542, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:22.375511 29764 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:12:22.376219 29764 webserver.cc:492] Webserver started at http://127.14.212.65:38967/ using document root <none> and password file <none>
I20251026 15:12:22.376492 29764 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:22.376559 29764 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:22.378257 29764 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:12:22.379137 29799 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.379390 29764 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:12:22.379458 29764 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "3717f5812abb4c8cae8c7f1ca081ee69"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "773c601391b75ed3869536b2ad2bcfba"
server_key_iv: "36b4acbf513c15ea5061d1d4a3d46bf8"
server_key_version: "encryptionkey@0"
I20251026 15:12:22.379762 29764 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:22.407691 29764 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:22.410008 29764 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:22.410121 29764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:22.410470 29764 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:22.410920 29806 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251026 15:12:22.412101 29764 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 15:12:22.412158 29764 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 15:12:22.412196 29764 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251026 15:12:22.412770 29764 ts_tablet_manager.cc:616] Registered 1 tablets
I20251026 15:12:22.412802 29764 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251026 15:12:22.412894 29806 tablet_bootstrap.cc:492] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Bootstrap starting.
I20251026 15:12:22.420717 29764 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:44075
I20251026 15:12:22.421096 29764 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:12:22.422603 29913 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:44075 every 8 connection(s)
I20251026 15:12:22.425282 29806 log.cc:826] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:22.427140 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 29764
I20251026 15:12:22.427476 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:35933
--local_ip_for_outbound_sockets=127.14.212.66
--tserver_master_addrs=127.14.212.126:32837
--webserver_port=33563
--webserver_interface=127.14.212.66
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:12:22.429093 29806 tablet_bootstrap.cc:492] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 15:12:22.429561 29806 tablet_bootstrap.cc:492] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Bootstrap complete.
Oct 26 15:12:22 dist-test-slave-8pgz krb5kdc[29136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491542, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:22.431114 29806 ts_tablet_manager.cc:1403] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.003s
I20251026 15:12:22.432338 29806 raft_consensus.cc:359] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } } peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } }
I20251026 15:12:22.432744 29806 raft_consensus.cc:740] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3717f5812abb4c8cae8c7f1ca081ee69, State: Initialized, Role: FOLLOWER
I20251026 15:12:22.432893 29806 consensus_queue.cc:260] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } } peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } }
I20251026 15:12:22.433178 29806 ts_tablet_manager.cc:1434] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:12:22.435359 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.421886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:35925)
1026 15:12:22.422846 (+ 960us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.422874 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.423406 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.426306 (+ 2900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.426310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.426732 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:22.426997 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.427002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.428167 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.428169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.428748 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.428750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.428895 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.429545 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:22.429569 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:22.431505 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:22.433969 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:22.433975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:22.433977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:22.434177 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:22.434388 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:22.434415 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:22.434416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:22.434482 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:22.434798 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:22.434803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:22.434921 (+ 118us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.435008 (+ 87us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.435158 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":82,"threads_started":1}
I20251026 15:12:22.435359 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.421951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:35925 (local address 127.14.212.126:32837)
1026 15:12:22.425929 (+ 3978us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.425940 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.425966 (+ 26us) server_negotiation.cc:408] Connection header received
1026 15:12:22.426079 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.426084 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.426125 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.426266 (+ 141us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:22.427126 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.428058 (+ 932us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.429174 (+ 1116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.431647 (+ 2473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.431767 (+ 120us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:22.431789 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:22.431789 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:22.431803 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:22.433878 (+ 2075us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:22.434266 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:22.434268 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:22.434269 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:22.434325 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:22.434547 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:22.434548 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:22.434549 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:22.434725 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:22.434811 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.435087 (+ 276us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.435206 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3931,"thread_start_us":79,"threads_started":1}
I20251026 15:12:22.436401 29914 heartbeater.cc:344] Connected to a master server at 127.14.212.126:32837
I20251026 15:12:22.436502 29914 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:22.436702 29914 heartbeater.cc:507] Master 127.14.212.126:32837 requested a full tablet report, sending...
I20251026 15:12:22.437325 29726 ts_manager.cc:194] Registered new tserver with Master: 3717f5812abb4c8cae8c7f1ca081ee69 (127.14.212.65:44075)
I20251026 15:12:22.438382 29726 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:35925
I20251026 15:12:22.439188 29914 heartbeater.cc:499] Master 127.14.212.126:32837 was elected leader, sending a full tablet report...
I20251026 15:12:22.444854 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.437847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:44075 (local address 127.0.0.1:46874)
1026 15:12:22.442530 (+ 4683us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.442557 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.442640 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.442877 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.442879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.442881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:22.443104 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.443105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.443943 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.443945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.444566 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.444568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.444639 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.444681 (+ 42us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.444730 (+ 49us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.444769 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4653,"thread_start_us":99,"threads_started":1}
I20251026 15:12:22.445335 29922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.438473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.14.212.65:44075)
1026 15:12:22.442744 (+ 4271us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.442750 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.442757 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:12:22.442782 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.442783 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.442796 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.442853 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:22.443179 (+ 326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.443868 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.444649 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.445161 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.445217 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.445248 (+ 31us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.445287 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4246,"thread_start_us":99,"threads_started":1}
I20251026 15:12:22.445623 29848 tablet_service.cc:1552] Processing DeleteTablet for tablet 07f44eb3a34845df84aaec042cbc0327 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:22 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:46874
I20251026 15:12:22.446062 29923 tablet_replica.cc:333] stopping tablet replica
I20251026 15:12:22.446148 29923 raft_consensus.cc:2243] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:12:22.446223 29923 raft_consensus.cc:2272] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:12:22.446532 29923 ts_tablet_manager.cc:1916] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:12:22.448202 29923 ts_tablet_manager.cc:1929] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:12:22.448266 29923 log.cc:1199] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal/wals/07f44eb3a34845df84aaec042cbc0327
I20251026 15:12:22.448539 29923 ts_tablet_manager.cc:1950] T 07f44eb3a34845df84aaec042cbc0327 P 3717f5812abb4c8cae8c7f1ca081ee69: Deleting consensus metadata
I20251026 15:12:22.448987 29710 catalog_manager.cc:4985] TS 3717f5812abb4c8cae8c7f1ca081ee69 (127.14.212.65:44075): tablet 07f44eb3a34845df84aaec042cbc0327 (table test-table [id=a1d673fd89c14a7db09ef29df2d760be]) successfully deleted
W20251026 15:12:22.531534 29919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:22.531733 29919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:22.531759 29919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:22.533262 29919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:22.533316 29919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:22.533345 29919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:12:22.535077 29919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:35933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=33563
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:32837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:22.535298 29919 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:22.535506 29919 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:22.538511 29929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:22.538511 29931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:22.538561 29919 server_base.cc:1047] running on GCE node
W20251026 15:12:22.538514 29928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:22.538846 29919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:22.539042 29919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:22.540181 29919 hybrid_clock.cc:648] HybridClock initialized: now 1761491542540161 us; error 30 us; skew 500 ppm
Oct 26 15:12:22 dist-test-slave-8pgz krb5kdc[29136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491542, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:22.542856 29919 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:12:22.543432 29919 webserver.cc:492] Webserver started at http://127.14.212.66:33563/ using document root <none> and password file <none>
I20251026 15:12:22.543628 29919 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:22.543673 29919 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:22.544878 29919 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:22.545399 29938 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.545580 29919 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.545657 29919 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "54d141fdf2874e50b643d9dafaf62410"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "2d9e6e0cd91423a4229285946303524f"
server_key_iv: "b3ef628c0d497d8d4a34ba1f17d9521c"
server_key_version: "encryptionkey@0"
I20251026 15:12:22.545912 29919 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:22.564028 29919 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:22.565987 29919 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:22.566097 29919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:22.566284 29919 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:22.566663 29919 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:22.566700 29919 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.566725 29919 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:22.566752 29919 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.572695 29919 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:35933
I20251026 15:12:22.572772 30051 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:35933 every 8 connection(s)
I20251026 15:12:22.573053 29919 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:12:22.573781 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 29919
I20251026 15:12:22.574131 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:37101
--local_ip_for_outbound_sockets=127.14.212.67
--tserver_master_addrs=127.14.212.126:32837
--webserver_port=35515
--webserver_interface=127.14.212.67
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 26 15:12:22 dist-test-slave-8pgz krb5kdc[29136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491542, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:22.584182 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.573816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:49389 (local address 127.14.212.126:32837)
1026 15:12:22.573885 (+ 69us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.573887 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.574079 (+ 192us) server_negotiation.cc:408] Connection header received
1026 15:12:22.574883 (+ 804us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.574885 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.574899 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.574958 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:22.576103 (+ 1145us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.576937 (+ 834us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.577657 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.577884 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.580691 (+ 2807us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:22.580698 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:22.580699 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:22.580709 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:22.582668 (+ 1959us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:22.583093 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:22.583095 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:22.583096 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:22.583126 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:22.583428 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:22.583429 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:22.583430 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:22.583561 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:22.583650 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.583951 (+ 301us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.584056 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251026 15:12:22.584235 30054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.573786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:49389)
1026 15:12:22.573947 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.573985 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.574774 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.575037 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.575041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.575560 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:22.575994 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.575999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.577061 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.577062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.577520 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.577523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.577692 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.578751 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:22.578772 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:22.580575 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:22.582762 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:22.582769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:22.582772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:22.583005 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:22.583220 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:22.583222 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:22.583223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:22.583355 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:22.583662 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:22.583668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:22.583811 (+ 143us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.583914 (+ 103us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.584049 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20251026 15:12:22.584846 30052 heartbeater.cc:344] Connected to a master server at 127.14.212.126:32837
I20251026 15:12:22.584964 30052 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:22.585232 30052 heartbeater.cc:507] Master 127.14.212.126:32837 requested a full tablet report, sending...
I20251026 15:12:22.585711 29726 ts_manager.cc:194] Registered new tserver with Master: 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933)
I20251026 15:12:22.586256 29726 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:49389
W20251026 15:12:22.658303 30055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:22.658574 30055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:22.658604 30055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:22.660167 30055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:22.660223 30055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:22.660251 30055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:12:22.661932 30055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:37101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=35515
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:32837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:22.662163 30055 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:22.662376 30055 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:22.665122 30061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:22.665132 30063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:22.665253 30060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:22.665326 30055 server_base.cc:1047] running on GCE node
I20251026 15:12:22.665656 30055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:22.665872 30055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:22.667042 30055 hybrid_clock.cc:648] HybridClock initialized: now 1761491542667018 us; error 39 us; skew 500 ppm
Oct 26 15:12:22 dist-test-slave-8pgz krb5kdc[29136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491542, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:22.669775 30055 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:12:22.670291 30055 webserver.cc:492] Webserver started at http://127.14.212.67:35515/ using document root <none> and password file <none>
I20251026 15:12:22.670506 30055 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:22.670562 30055 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:22.671864 30055 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:12:22.672482 30070 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.672706 30055 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:12:22.672775 30055 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "d2be86aaf16042299571a85097485a13"
format_stamp: "Formatted at 2025-10-26 15:12:22 on dist-test-slave-8pgz"
server_key: "d305f831ccbc6c3c47800ec2534d43e2"
server_key_iv: "d51cdb92175fa0a39440c66eea9ee634"
server_key_version: "encryptionkey@0"
I20251026 15:12:22.673028 30055 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:22.689076 30055 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:22.691345 30055 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:22.691463 30055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:22.691650 30055 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:22.691957 30055 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:22.691994 30055 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.692018 30055 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:22.692030 30055 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.697623 30055 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:37101
I20251026 15:12:22.697763 30183 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:37101 every 8 connection(s)
I20251026 15:12:22.698009 30055 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:12:22.699488 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 30055
Oct 26 15:12:22 dist-test-slave-8pgz krb5kdc[29136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491542, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:22.707165 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.699947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.14.212.126:32837)
1026 15:12:22.700044 (+ 97us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.700046 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.700160 (+ 114us) server_negotiation.cc:408] Connection header received
1026 15:12:22.700187 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.700188 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.700199 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.700263 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:22.701127 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.701854 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.702737 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.702865 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.704052 (+ 1187us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:22.704057 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:22.704057 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:22.704063 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:22.705703 (+ 1640us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:22.706215 (+ 512us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:22.706217 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:22.706217 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:22.706240 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:22.706498 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:22.706499 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:22.706499 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:22.706594 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:22.706759 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.706950 (+ 191us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.707040 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":53,"threads_started":1}
I20251026 15:12:22.708575 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.698790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:34147 (local address 127.14.212.126:32837)
1026 15:12:22.698851 (+ 61us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.698853 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.699024 (+ 171us) server_negotiation.cc:408] Connection header received
1026 15:12:22.699753 (+ 729us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.699755 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.699770 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.699849 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:22.701048 (+ 1199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.701876 (+ 828us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.703123 (+ 1247us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.703295 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.705815 (+ 2520us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:22.705821 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:22.705821 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:22.705829 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:22.707287 (+ 1458us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:22.707691 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:22.707692 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:22.707692 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:22.707719 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:22.707892 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:22.707892 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:22.707893 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:22.707998 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:22.708067 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.708431 (+ 364us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.708515 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251026 15:12:22.708678 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.698746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:34147)
1026 15:12:22.698926 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.698956 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.699596 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.699962 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.699968 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.700524 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:22.700951 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.700956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.702495 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.702496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.703042 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.703045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.703190 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.704161 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:22.704177 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:22.705729 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:22.707379 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:22.707387 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:22.707389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:22.707627 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:22.707786 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:22.707786 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:22.707787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:22.707842 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:22.708091 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:22.708097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:22.708237 (+ 140us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.708382 (+ 145us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.708497 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":68,"threads_started":1}
I20251026 15:12:22.709195 30184 heartbeater.cc:344] Connected to a master server at 127.14.212.126:32837
I20251026 15:12:22.709286 30184 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:22.709528 30184 heartbeater.cc:507] Master 127.14.212.126:32837 requested a full tablet report, sending...
I20251026 15:12:22.709933 29726 ts_manager.cc:194] Registered new tserver with Master: d2be86aaf16042299571a85097485a13 (127.14.212.67:37101)
I20251026 15:12:22.710297 29726 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:34147
I20251026 15:12:22.719116 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:12:22.722179 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.719476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.14.212.126:32837)
1026 15:12:22.719551 (+ 75us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.719553 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.719662 (+ 109us) server_negotiation.cc:408] Connection header received
1026 15:12:22.719738 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.719739 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.719764 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.719808 (+ 44us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:12:22.720075 (+ 267us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.720663 (+ 588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.721543 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.721732 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.721789 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:12:22.721923 (+ 134us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:12:22.722014 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.722058 (+ 44us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.722091 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251026 15:12:22.722566 29726 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:12:22.724261 29726 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:12:22.726385 29848 tablet_service.cc:1505] Processing CreateTablet for tablet f6a15be46af34060ac9be51d945b5302 (DEFAULT_TABLE table=test-table [id=61c8f24575de4324a56d4ab7b9300981]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:12:22.726632 29848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a15be46af34060ac9be51d945b5302. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:22.728206 29806 tablet_bootstrap.cc:492] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69: Bootstrap starting.
I20251026 15:12:22.728586 29806 tablet_bootstrap.cc:654] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:22.728621 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.725962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:35933 (local address 127.0.0.1:49854)
1026 15:12:22.726018 (+ 56us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.726031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.726077 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.726469 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.726470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.726473 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:22.726686 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.726688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.727721 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.727723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.728385 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.728388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.728462 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.728466 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.728500 (+ 34us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.728547 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20251026 15:12:22.729163 29806 tablet_bootstrap.cc:492] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69: No bootstrap required, opened a new log
I20251026 15:12:22.729219 29806 ts_tablet_manager.cc:1403] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 15:12:22.729337 29806 raft_consensus.cc:359] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:22.729411 29806 raft_consensus.cc:385] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:22.729451 29806 raft_consensus.cc:740] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3717f5812abb4c8cae8c7f1ca081ee69, State: Initialized, Role: FOLLOWER
I20251026 15:12:22.729499 29806 consensus_queue.cc:260] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:22.729590 29806 ts_tablet_manager.cc:1434] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.729655 30191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.726087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.14.212.66:35933)
1026 15:12:22.726267 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.726275 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.726286 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:12:22.726330 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.726332 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.726351 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.726466 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:22.726767 (+ 301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.727649 (+ 882us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.728739 (+ 1090us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.729389 (+ 650us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.729455 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.729490 (+ 35us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.729536 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20251026 15:12:22.729676 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.726035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:37101 (local address 127.0.0.1:44806)
1026 15:12:22.726159 (+ 124us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.726167 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.726219 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.727058 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.727059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.727062 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:22.727268 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.727270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.728476 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.728477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.729461 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.729464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.729529 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.729533 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.729574 (+ 41us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.729614 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20251026 15:12:22.730273 29986 tablet_service.cc:1505] Processing CreateTablet for tablet f6a15be46af34060ac9be51d945b5302 (DEFAULT_TABLE table=test-table [id=61c8f24575de4324a56d4ab7b9300981]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:12:22.730552 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.726312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.14.212.67:37101)
1026 15:12:22.726841 (+ 529us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.726848 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.726858 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:12:22.726896 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.726898 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.726913 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.726981 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:22.727329 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.728389 (+ 1060us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.729790 (+ 1401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.730332 (+ 542us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.730385 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.730434 (+ 49us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.730477 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":144,"threads_started":1}
I20251026 15:12:22.730640 29986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a15be46af34060ac9be51d945b5302. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:22.731057 30118 tablet_service.cc:1505] Processing CreateTablet for tablet f6a15be46af34060ac9be51d945b5302 (DEFAULT_TABLE table=test-table [id=61c8f24575de4324a56d4ab7b9300981]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:12:22.731310 30118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a15be46af34060ac9be51d945b5302. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:22.733830 30194 tablet_bootstrap.cc:492] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410: Bootstrap starting.
I20251026 15:12:22.734192 30195 tablet_bootstrap.cc:492] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13: Bootstrap starting.
I20251026 15:12:22.734575 30194 tablet_bootstrap.cc:654] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:22.734812 30195 tablet_bootstrap.cc:654] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:22.734843 30194 log.cc:826] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:22.735069 30195 log.cc:826] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:22.735531 30194 tablet_bootstrap.cc:492] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410: No bootstrap required, opened a new log
I20251026 15:12:22.735589 30194 ts_tablet_manager.cc:1403] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 15:12:22.735702 30195 tablet_bootstrap.cc:492] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13: No bootstrap required, opened a new log
I20251026 15:12:22.735762 30195 ts_tablet_manager.cc:1403] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:12:22.736759 30194 raft_consensus.cc:359] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:22.736889 30194 raft_consensus.cc:385] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:22.736927 30194 raft_consensus.cc:740] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d141fdf2874e50b643d9dafaf62410, State: Initialized, Role: FOLLOWER
I20251026 15:12:22.736816 30195 raft_consensus.cc:359] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:22.736963 30195 raft_consensus.cc:385] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:22.737033 30195 raft_consensus.cc:740] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2be86aaf16042299571a85097485a13, State: Initialized, Role: FOLLOWER
I20251026 15:12:22.737049 30194 consensus_queue.cc:260] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:22.737164 30195 consensus_queue.cc:260] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:22.737368 30194 ts_tablet_manager.cc:1434] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:12:22.737390 30052 heartbeater.cc:499] Master 127.14.212.126:32837 was elected leader, sending a full tablet report...
I20251026 15:12:22.737447 30195 ts_tablet_manager.cc:1434] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:12:22.737483 30184 heartbeater.cc:499] Master 127.14.212.126:32837 was elected leader, sending a full tablet report...
I20251026 15:12:22.791253 30199 raft_consensus.cc:493] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:12:22.791388 30199 raft_consensus.cc:515] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:22.791709 30199 leader_election.cc:290] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933), 3717f5812abb4c8cae8c7f1ca081ee69 (127.14.212.65:44075)
I20251026 15:12:22.794051 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.791889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:35933 (local address 127.14.212.67:57545)
1026 15:12:22.791947 (+ 58us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.791964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.792035 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.792196 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.792197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.792200 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:22.792475 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.792477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.793210 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.793212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.793810 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.793812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.793882 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.793887 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.793920 (+ 33us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.793957 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20251026 15:12:22.794063 30200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.791994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:44075 (local address 127.14.212.67:50007)
1026 15:12:22.792176 (+ 182us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.792185 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:22.792239 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:22.792375 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:22.792375 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:22.792386 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:22.792620 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.792621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.793233 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.793234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:22.793822 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:22.793823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.793894 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.793898 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:12:22.793930 (+ 32us) client_negotiation.cc:240] Negotiation successful
1026 15:12:22.793965 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":99,"threads_started":1}
I20251026 15:12:22.794533 29922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.792096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:50007 (local address 127.14.212.65:44075)
1026 15:12:22.792145 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.792147 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.792252 (+ 105us) server_negotiation.cc:408] Connection header received
1026 15:12:22.792302 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.792303 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.792312 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.792368 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:22.792702 (+ 334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.793149 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.793907 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.794423 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.794433 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.794459 (+ 26us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.794487 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251026 15:12:22.794530 30191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.791981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:57545 (local address 127.14.212.66:35933)
1026 15:12:22.792032 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.792034 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.792046 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:12:22.792114 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.792115 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.792126 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.792184 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:22.792583 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.793117 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.793900 (+ 783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.794411 (+ 511us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.794422 (+ 11us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.794446 (+ 24us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.794483 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251026 15:12:22.794822 30006 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6a15be46af34060ac9be51d945b5302" candidate_uuid: "d2be86aaf16042299571a85097485a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d141fdf2874e50b643d9dafaf62410" is_pre_election: true
I20251026 15:12:22.794817 29868 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6a15be46af34060ac9be51d945b5302" candidate_uuid: "d2be86aaf16042299571a85097485a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" is_pre_election: true
I20251026 15:12:22.794960 30006 raft_consensus.cc:2468] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2be86aaf16042299571a85097485a13 in term 0.
I20251026 15:12:22.794960 29868 raft_consensus.cc:2468] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2be86aaf16042299571a85097485a13 in term 0.
I20251026 15:12:22.795198 30071 leader_election.cc:304] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54d141fdf2874e50b643d9dafaf62410, d2be86aaf16042299571a85097485a13; no voters:
I20251026 15:12:22.795353 30199 raft_consensus.cc:2804] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:12:22.795420 30199 raft_consensus.cc:493] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:12:22.795439 30199 raft_consensus.cc:3060] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:22.796121 30199 raft_consensus.cc:515] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:22.796238 30199 leader_election.cc:290] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [CANDIDATE]: Term 1 election: Requested vote from peers 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933), 3717f5812abb4c8cae8c7f1ca081ee69 (127.14.212.65:44075)
I20251026 15:12:22.796538 29868 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6a15be46af34060ac9be51d945b5302" candidate_uuid: "d2be86aaf16042299571a85097485a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3717f5812abb4c8cae8c7f1ca081ee69"
I20251026 15:12:22.796550 30006 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f6a15be46af34060ac9be51d945b5302" candidate_uuid: "d2be86aaf16042299571a85097485a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d141fdf2874e50b643d9dafaf62410"
I20251026 15:12:22.796624 30006 raft_consensus.cc:3060] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:22.796625 29868 raft_consensus.cc:3060] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:22.797230 29868 raft_consensus.cc:2468] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2be86aaf16042299571a85097485a13 in term 1.
I20251026 15:12:22.797312 30006 raft_consensus.cc:2468] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2be86aaf16042299571a85097485a13 in term 1.
I20251026 15:12:22.797444 30071 leader_election.cc:304] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3717f5812abb4c8cae8c7f1ca081ee69, d2be86aaf16042299571a85097485a13; no voters:
I20251026 15:12:22.797544 30199 raft_consensus.cc:2804] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:12:22.797700 30199 raft_consensus.cc:697] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [term 1 LEADER]: Becoming Leader. State: Replica: d2be86aaf16042299571a85097485a13, State: Running, Role: LEADER
I20251026 15:12:22.797788 30199 consensus_queue.cc:237] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:22.798606 29725 catalog_manager.cc:5649] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2be86aaf16042299571a85097485a13 (127.14.212.67). New cstate: current_term: 1 leader_uuid: "d2be86aaf16042299571a85097485a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } health_report { overall_health: UNKNOWN } } }
I20251026 15:12:22.803821 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.801851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.14.212.67:37101)
1026 15:12:22.801923 (+ 72us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:22.801924 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:22.801933 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:12:22.801963 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:22.801963 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:22.801978 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:22.802029 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:12:22.802234 (+ 205us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.802747 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:22.803271 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:22.803381 (+ 110us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:22.803410 (+ 29us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:12:22.803499 (+ 89us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:12:22.803589 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:22.803741 (+ 152us) server_negotiation.cc:300] Negotiation successful
1026 15:12:22.803767 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251026 15:12:22.805420 29868 raft_consensus.cc:1275] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Refusing update from remote peer d2be86aaf16042299571a85097485a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:12:22.805495 30006 raft_consensus.cc:1275] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410 [term 1 FOLLOWER]: Refusing update from remote peer d2be86aaf16042299571a85097485a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:12:22.805778 30201 consensus_queue.cc:1048] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:12:22.805894 30199 consensus_queue.cc:1048] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:12:22.810207 30205 mvcc.cc:204] Tried to move back new op lower bound from 7215069359328497664 to 7215069359300370432. Current Snapshot: MvccSnapshot[applied={T|T < 7215069359328497664 or (T in {7215069359328497664})}]
I20251026 15:12:22.810441 29724 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38234:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:12:22.810541 29724 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38234:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:12:22.811267 29724 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Sending DeleteTablet for 3 replicas of tablet f6a15be46af34060ac9be51d945b5302
I20251026 15:12:22.811609 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 29764
I20251026 15:12:22.811582 29986 tablet_service.cc:1552] Processing DeleteTablet for tablet f6a15be46af34060ac9be51d945b5302 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:22 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:49854
I20251026 15:12:22.811638 29848 tablet_service.cc:1552] Processing DeleteTablet for tablet f6a15be46af34060ac9be51d945b5302 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:22 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:46874
I20251026 15:12:22.811723 29923 tablet_replica.cc:333] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69: stopping tablet replica
I20251026 15:12:22.811769 29923 raft_consensus.cc:2243] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:12:22.811826 29923 raft_consensus.cc:2272] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:12:22.811973 30118 tablet_service.cc:1552] Processing DeleteTablet for tablet f6a15be46af34060ac9be51d945b5302 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:22 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:44806
I20251026 15:12:22.812187 30215 tablet_replica.cc:333] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13: stopping tablet replica
I20251026 15:12:22.812250 30214 tablet_replica.cc:333] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410: stopping tablet replica
I20251026 15:12:22.812266 30215 raft_consensus.cc:2243] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:12:22.812328 30214 raft_consensus.cc:2243] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:12:22.812352 30215 raft_consensus.cc:2272] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:12:22.812423 30214 raft_consensus.cc:2272] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:12:22.812752 30215 ts_tablet_manager.cc:1916] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:12:22.812852 30214 ts_tablet_manager.cc:1916] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:12:22.814549 30215 ts_tablet_manager.cc:1929] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:12:22.814625 30215 log.cc:1199] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/f6a15be46af34060ac9be51d945b5302
I20251026 15:12:22.814862 30214 ts_tablet_manager.cc:1929] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:12:22.814916 30215 ts_tablet_manager.cc:1950] T f6a15be46af34060ac9be51d945b5302 P d2be86aaf16042299571a85097485a13: Deleting consensus metadata
I20251026 15:12:22.814926 30214 log.cc:1199] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/f6a15be46af34060ac9be51d945b5302
I20251026 15:12:22.815223 30214 ts_tablet_manager.cc:1950] T f6a15be46af34060ac9be51d945b5302 P 54d141fdf2874e50b643d9dafaf62410: Deleting consensus metadata
I20251026 15:12:22.815557 29710 catalog_manager.cc:4985] TS d2be86aaf16042299571a85097485a13 (127.14.212.67:37101): tablet f6a15be46af34060ac9be51d945b5302 (table test-table [id=61c8f24575de4324a56d4ab7b9300981]) successfully deleted
I20251026 15:12:22.815671 29710 catalog_manager.cc:4985] TS 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933): tablet f6a15be46af34060ac9be51d945b5302 (table test-table [id=61c8f24575de4324a56d4ab7b9300981]) successfully deleted
W20251026 15:12:22.818871 29710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:44075 (error 108)
I20251026 15:12:22.819262 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 29919
I20251026 15:12:22.823639 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:22.823464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:44075 (local address 127.0.0.1:46878)
1026 15:12:22.823526 (+ 62us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:22.823581 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:44075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20251026 15:12:22.823838 29710 catalog_manager.cc:4712] TS 3717f5812abb4c8cae8c7f1ca081ee69 (127.14.212.65:44075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6a15be46af34060ac9be51d945b5302: Network error: Client connection negotiation failed: client connection to 127.14.212.65:44075: connect: Connection refused (error 111)
I20251026 15:12:22.824960 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 30055
I20251026 15:12:22.829540 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 29693
I20251026 15:12:22.835129 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:32837
--webserver_interface=127.14.212.126
--webserver_port=35029
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:32837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:22.915243 30217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:22.915463 30217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:22.915504 30217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:22.916982 30217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:12:22.917039 30217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:22.917061 30217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:22.917083 30217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:12:22.917101 30217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:12:22.918888 30217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:32837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:32837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=35029
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:22.919180 30217 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:22.919382 30217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:22.922286 30225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:22.922449 30222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:22.922549 30223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:22.922449 30217 server_base.cc:1047] running on GCE node
I20251026 15:12:22.922848 30217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:22.923100 30217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:22.924271 30217 hybrid_clock.cc:648] HybridClock initialized: now 1761491542924248 us; error 46 us; skew 500 ppm
Oct 26 15:12:22 dist-test-slave-8pgz krb5kdc[29136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491542, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:22.927017 30217 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:12:22.927647 30217 webserver.cc:492] Webserver started at http://127.14.212.126:35029/ using document root <none> and password file <none>
I20251026 15:12:22.927856 30217 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:22.927896 30217 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:22.929186 30217 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:22.929808 30232 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:22.930064 30217 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:22.930145 30217 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "a1abd0e586c848209053268370498ae8"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "b23caa226e4d39f0c7c6f25459aab7af"
server_key_iv: "49d83dadc66c053e5b7c59897e558d17"
server_key_version: "encryptionkey@0"
I20251026 15:12:22.930464 30217 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:22.948144 30217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:22.950176 30217 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:22.950309 30217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:22.954304 30217 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:32837
I20251026 15:12:22.954370 30284 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:32837 every 8 connection(s)
I20251026 15:12:22.954717 30217 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:12:22.955880 30285 sys_catalog.cc:263] Verifying existing consensus state
I20251026 15:12:22.956435 30285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Bootstrap starting.
I20251026 15:12:22.959890 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 30217
I20251026 15:12:22.960228 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:44075
--local_ip_for_outbound_sockets=127.14.212.65
--tserver_master_addrs=127.14.212.126:32837
--webserver_port=38967
--webserver_interface=127.14.212.65
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:12:22.966796 30285 log.cc:826] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:22.969053 30285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 15:12:22.969360 30285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8: Bootstrap complete.
I20251026 15:12:22.971555 30285 raft_consensus.cc:359] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } }
I20251026 15:12:22.971776 30285 raft_consensus.cc:740] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1abd0e586c848209053268370498ae8, State: Initialized, Role: FOLLOWER
I20251026 15:12:22.971902 30285 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } }
I20251026 15:12:22.971971 30285 raft_consensus.cc:399] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:12:22.972023 30285 raft_consensus.cc:493] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:12:22.972062 30285 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 2 FOLLOWER]: Advancing to term 3
I20251026 15:12:22.973013 30285 raft_consensus.cc:515] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } }
I20251026 15:12:22.973117 30285 leader_election.cc:304] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1abd0e586c848209053268370498ae8; no voters:
I20251026 15:12:22.973282 30285 leader_election.cc:290] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20251026 15:12:22.973368 30289 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 15:12:22.973551 30285 sys_catalog.cc:565] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:12:22.973569 30289 raft_consensus.cc:697] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [term 3 LEADER]: Becoming Leader. State: Replica: a1abd0e586c848209053268370498ae8, State: Running, Role: LEADER
I20251026 15:12:22.973714 30289 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } }
I20251026 15:12:22.974010 30291 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1abd0e586c848209053268370498ae8. Latest consensus state: current_term: 3 leader_uuid: "a1abd0e586c848209053268370498ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } } }
I20251026 15:12:22.974140 30291 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:22.974330 30290 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a1abd0e586c848209053268370498ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1abd0e586c848209053268370498ae8" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 32837 } } }
I20251026 15:12:22.974375 30294 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:12:22.975219 30294 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:12:22.975356 30294 catalog_manager.cc:1269] Loaded cluster ID: 858aaa487627451483b66409cdc05e25
I20251026 15:12:22.975390 30294 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
F20251026 15:12:22.975612 30294 catalog_manager.cc:1476] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f63fc83adcd google::LogMessage::Fail() at ??:0
I20251026 15:12:22.974498 30290 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1abd0e586c848209053268370498ae8 [sys.catalog]: This master's current role is: LEADER
@ 0x7f63fc83eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f63fc83a7cc google::LogMessage::Flush() at ??:0
@ 0x7f63fc83bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f63ff088bd6 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f63ff093334 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f63fd93c3a5 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f63fd9325d7 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f63fdc676db start_thread at ??:0
@ 0x7f63fbd8f71f clone at ??:0
W20251026 15:12:23.074008 30287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:23.074222 30287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:23.074265 30287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:23.075839 30287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:23.075891 30287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:23.075933 30287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:12:23.077550 30287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:44075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=38967
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:32837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:23.077816 30287 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:23.077992 30287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:23.080821 30322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:23.080962 30287 server_base.cc:1047] running on GCE node
W20251026 15:12:23.080835 30324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:23.081058 30321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:23.081219 30287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:23.081427 30287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:23.082571 30287 hybrid_clock.cc:648] HybridClock initialized: now 1761491543082557 us; error 31 us; skew 500 ppm
Oct 26 15:12:23 dist-test-slave-8pgz krb5kdc[29136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:23.085234 30287 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:12:23.085767 30287 webserver.cc:492] Webserver started at http://127.14.212.65:38967/ using document root <none> and password file <none>
I20251026 15:12:23.085963 30287 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:23.086004 30287 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:23.087340 30287 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:23.087999 30331 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:23.088181 30287 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:12:23.088253 30287 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "3717f5812abb4c8cae8c7f1ca081ee69"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "773c601391b75ed3869536b2ad2bcfba"
server_key_iv: "36b4acbf513c15ea5061d1d4a3d46bf8"
server_key_version: "encryptionkey@0"
I20251026 15:12:23.088543 30287 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:23.103175 30287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:23.105237 30287 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:23.105391 30287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:23.105639 30287 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:23.106129 30338 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251026 15:12:23.107249 30287 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 15:12:23.107295 30287 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 15:12:23.107333 30287 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251026 15:12:23.107892 30287 ts_tablet_manager.cc:616] Registered 1 tablets
I20251026 15:12:23.107924 30287 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251026 15:12:23.108000 30338 tablet_bootstrap.cc:492] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69: Bootstrap starting.
I20251026 15:12:23.115394 30287 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:44075
I20251026 15:12:23.115450 30445 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:44075 every 8 connection(s)
I20251026 15:12:23.115784 30287 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:12:23.120745 30338 log.cc:826] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:23.123283 30338 tablet_bootstrap.cc:492] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 15:12:23.123646 30338 tablet_bootstrap.cc:492] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69: Bootstrap complete.
I20251026 15:12:23.124900 30338 ts_tablet_manager.cc:1403] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20251026 15:12:23.125865 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 30287
I20251026 15:12:23.126143 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:35933
--local_ip_for_outbound_sockets=127.14.212.66
--tserver_master_addrs=127.14.212.126:32837
--webserver_port=33563
--webserver_interface=127.14.212.66
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:12:23.126104 30338 raft_consensus.cc:359] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:23.126368 30338 raft_consensus.cc:740] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3717f5812abb4c8cae8c7f1ca081ee69, State: Initialized, Role: FOLLOWER
I20251026 15:12:23.126524 30338 consensus_queue.cc:260] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:23.126794 30338 ts_tablet_manager.cc:1434] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20251026 15:12:23.208745 30448 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:12:23.116451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:33129)
1026 15:12:23.116623 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:23.116653 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:23.117102 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:23.208587 (+ 91485us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":107,"threads_started":1}
W20251026 15:12:23.209039 30446 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:32837 (0 consecutive failures): Network error: Failed to ping master at 127.14.212.126:32837: Client connection negotiation failed: client connection to 127.14.212.126:32837: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251026 15:12:23.209015 30451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:23.209244 30451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:23.209298 30451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251026 15:12:23.209424 30448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:23.209310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:34125)
1026 15:12:23.209365 (+ 55us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:23.209399 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20251026 15:12:23.209753 30448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:23.209673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:37379)
1026 15:12:23.209704 (+ 31us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:23.209732 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20251026 15:12:23.209868 30446 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251026 15:12:23.211169 30451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:23.211220 30451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:23.211266 30451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:12:23.213107 30451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:35933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=33563
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:32837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:23.213362 30451 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:23.213552 30451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:23.216240 30456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:23.216291 30457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:23.216365 30451 server_base.cc:1047] running on GCE node
W20251026 15:12:23.216253 30459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:23.216606 30451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:23.216820 30451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:23.217972 30451 hybrid_clock.cc:648] HybridClock initialized: now 1761491543217943 us; error 50 us; skew 500 ppm
Oct 26 15:12:23 dist-test-slave-8pgz krb5kdc[29136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:23.220750 30451 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:12:23.221369 30451 webserver.cc:492] Webserver started at http://127.14.212.66:33563/ using document root <none> and password file <none>
I20251026 15:12:23.221578 30451 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:23.221621 30451 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:23.223084 30451 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 15:12:23.223696 30466 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:23.223870 30451 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:12:23.223938 30451 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "54d141fdf2874e50b643d9dafaf62410"
format_stamp: "Formatted at 2025-10-26 15:12:21 on dist-test-slave-8pgz"
server_key: "2d9e6e0cd91423a4229285946303524f"
server_key_iv: "b3ef628c0d497d8d4a34ba1f17d9521c"
server_key_version: "encryptionkey@0"
I20251026 15:12:23.224200 30451 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:23.232290 30451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:23.234251 30451 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:23.234455 30451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:23.234678 30451 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:23.234966 30451 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:23.234999 30451 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:23.235033 30451 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:23.235064 30451 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:23.240986 30451 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:35933
I20251026 15:12:23.241050 30579 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:35933 every 8 connection(s)
I20251026 15:12:23.241370 30451 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:12:23.242502 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:23.242091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:48673)
1026 15:12:23.242242 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:23.242323 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
W20251026 15:12:23.242731 30580 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:32837 (0 consecutive failures): Network error: Failed to ping master at 127.14.212.126:32837: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
I20251026 15:12:23.243079 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:23.242956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:56351)
1026 15:12:23.243024 (+ 68us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:23.243056 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20251026 15:12:23.243523 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:23.243367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:40569)
1026 15:12:23.243455 (+ 88us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:23.243493 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20251026 15:12:23.243639 30580 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251026 15:12:23.251484 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 30451
I20251026 15:12:23.251756 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:37101
--local_ip_for_outbound_sockets=127.14.212.67
--tserver_master_addrs=127.14.212.126:32837
--webserver_port=35515
--webserver_interface=127.14.212.67
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:23.334689 30583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:23.334896 30583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:23.334939 30583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:23.336443 30583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:23.336505 30583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:23.336562 30583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:12:23.338320 30583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:37101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=35515
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:32837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:23.338634 30583 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:23.338835 30583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:23.341558 30591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:23.341558 30589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:23.341656 30583 server_base.cc:1047] running on GCE node
W20251026 15:12:23.341583 30588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:23.342011 30583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:23.342211 30583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:23.343356 30583 hybrid_clock.cc:648] HybridClock initialized: now 1761491543343342 us; error 27 us; skew 500 ppm
Oct 26 15:12:23 dist-test-slave-8pgz krb5kdc[29136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:23.345796 30583 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:12:23.346323 30583 webserver.cc:492] Webserver started at http://127.14.212.67:35515/ using document root <none> and password file <none>
I20251026 15:12:23.346529 30583 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:23.346570 30583 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:23.347816 30583 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:23.348378 30598 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:23.348560 30583 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:12:23.348634 30583 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "d2be86aaf16042299571a85097485a13"
format_stamp: "Formatted at 2025-10-26 15:12:22 on dist-test-slave-8pgz"
server_key: "d305f831ccbc6c3c47800ec2534d43e2"
server_key_iv: "d51cdb92175fa0a39440c66eea9ee634"
server_key_version: "encryptionkey@0"
I20251026 15:12:23.348894 30583 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:23.360726 30583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:23.362753 30583 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:23.362896 30583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:23.363106 30583 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:23.363425 30583 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:23.363457 30583 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:23.363495 30583 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:23.363525 30583 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:23.369000 30583 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:37101
I20251026 15:12:23.369086 30711 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:37101 every 8 connection(s)
I20251026 15:12:23.369382 30583 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:12:23.370338 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:23.369967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:42703)
1026 15:12:23.370119 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:23.370195 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
W20251026 15:12:23.370572 30712 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:32837 (0 consecutive failures): Network error: Failed to ping master at 127.14.212.126:32837: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
I20251026 15:12:23.370826 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:23.370747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:37249)
1026 15:12:23.370786 (+ 39us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:23.370810 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20251026 15:12:23.371150 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:23.371013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:41975)
1026 15:12:23.371097 (+ 84us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:23.371127 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20251026 15:12:23.371299 30712 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251026 15:12:23.376807 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 30583
I20251026 15:12:24.210500 30716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:24.210232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:40317)
1026 15:12:24.210381 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:24.210445 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20251026 15:12:24.244218 30717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:24.243969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:52121)
1026 15:12:24.244116 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:24.244147 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20251026 15:12:24.371887 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:24.371668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:45457)
1026 15:12:24.371795 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:24.371834 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20251026 15:12:24.737811 30719 raft_consensus.cc:493] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:12:24.737939 30719 raft_consensus.cc:515] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:24.738327 30719 leader_election.cc:290] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933), d2be86aaf16042299571a85097485a13 (127.14.212.67:37101)
Oct 26 15:12:24 dist-test-slave-8pgz krb5kdc[29136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.67@KRBTEST.COM
Oct 26 15:12:24 dist-test-slave-8pgz krb5kdc[29136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491543, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.66@KRBTEST.COM
I20251026 15:12:24.748478 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:24.738755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:37101 (local address 127.14.212.65:37299)
1026 15:12:24.738839 (+ 84us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:24.738849 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:24.738913 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:24.739687 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:24.739690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:24.740180 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:24.740464 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:24.740468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:24.741845 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:24.741846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:24.742298 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:24.742300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:24.742381 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:24.742895 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:24.742900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:24.744604 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:24.746929 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:24.746937 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:24.746940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:24.747224 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:24.747509 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:24.747510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:24.747510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:24.747608 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:24.747981 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:24.747987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:24.748120 (+ 133us) client_negotiation.cc:769] Sending connection context
1026 15:12:24.748203 (+ 83us) client_negotiation.cc:240] Negotiation successful
1026 15:12:24.748366 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":37,"threads_started":1}
I20251026 15:12:24.748494 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:24.738904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:37299 (local address 127.14.212.67:37101)
1026 15:12:24.739026 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:24.739038 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:24.739053 (+ 15us) server_negotiation.cc:408] Connection header received
1026 15:12:24.739476 (+ 423us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:24.739478 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:24.739517 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:24.739613 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:24.740586 (+ 973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:24.741743 (+ 1157us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:24.742417 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:24.742708 (+ 291us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:24.744712 (+ 2004us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:24.744722 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:24.744722 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:24.744750 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:24.746842 (+ 2092us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:24.747367 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:24.747370 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:24.747370 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:24.747436 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:24.747678 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:24.747679 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:24.747679 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:24.747915 (+ 236us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:24.747988 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:24.748267 (+ 279us) server_negotiation.cc:300] Negotiation successful
1026 15:12:24.748382 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":53,"threads_started":1}
I20251026 15:12:24.748522 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:24.738625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:45745 (local address 127.14.212.66:35933)
1026 15:12:24.738785 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:24.738812 (+ 27us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:24.738842 (+ 30us) server_negotiation.cc:408] Connection header received
1026 15:12:24.739825 (+ 983us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:24.739829 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:24.739876 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:24.739998 (+ 122us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:24.740587 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:24.741284 (+ 697us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:24.742008 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:24.742286 (+ 278us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:24.745108 (+ 2822us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:24.745116 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:24.745117 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:24.745137 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:24.747190 (+ 2053us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:24.747555 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:24.747557 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:24.747558 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:24.747622 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:24.747767 (+ 145us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:24.747767 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:24.747767 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:24.747944 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:24.748033 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:24.748268 (+ 235us) server_negotiation.cc:300] Negotiation successful
1026 15:12:24.748430 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20251026 15:12:24.748605 30720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:24.738543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:35933 (local address 127.14.212.65:45745)
1026 15:12:24.738758 (+ 215us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:24.738772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:24.738850 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:24.740003 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:24.740004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:24.740186 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:24.740464 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:24.740467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:24.741387 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:24.741388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:24.741927 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:24.741930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:24.742025 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:24.742728 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:24.742762 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:24.745008 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:24.747283 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:24.747284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:24.747285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:24.747481 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:24.747682 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:24.747683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:24.747683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:24.747712 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:24.748014 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:24.748015 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:24.748105 (+ 90us) client_negotiation.cc:769] Sending connection context
1026 15:12:24.748203 (+ 98us) client_negotiation.cc:240] Negotiation successful
1026 15:12:24.748459 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":116,"thread_start_us":89,"threads_started":1}
W20251026 15:12:24.749974 30332 leader_election.cc:343] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933): Not found: Tablet not found: f6a15be46af34060ac9be51d945b5302
W20251026 15:12:24.750077 30332 leader_election.cc:343] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2be86aaf16042299571a85097485a13 (127.14.212.67:37101): Not found: Tablet not found: f6a15be46af34060ac9be51d945b5302
I20251026 15:12:24.750095 30332 leader_election.cc:304] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3717f5812abb4c8cae8c7f1ca081ee69; no voters: 54d141fdf2874e50b643d9dafaf62410, d2be86aaf16042299571a85097485a13
I20251026 15:12:24.750187 30719 raft_consensus.cc:2749] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 15:12:25.211191 30720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:25.211028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:32799)
1026 15:12:25.211101 (+ 73us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:25.211147 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20251026 15:12:25.244966 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:25.244713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:36897)
1026 15:12:25.244866 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:25.244908 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20251026 15:12:25.372735 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:25.372468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:48159)
1026 15:12:25.372640 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:25.372686 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
I20251026 15:12:26.212031 30726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:26.211796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:45813)
1026 15:12:26.211938 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:26.211979 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20251026 15:12:26.245813 30727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:26.245601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:47381)
1026 15:12:26.245728 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:26.245771 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20251026 15:12:26.373634 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:26.373371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:48389)
1026 15:12:26.373542 (+ 171us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:26.373587 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20251026 15:12:26.689934 30729 raft_consensus.cc:493] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:12:26.690030 30729 raft_consensus.cc:515] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:26.690191 30729 leader_election.cc:290] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933), d2be86aaf16042299571a85097485a13 (127.14.212.67:37101)
W20251026 15:12:26.690644 30332 leader_election.cc:343] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933): Not found: Tablet not found: f6a15be46af34060ac9be51d945b5302
W20251026 15:12:26.690764 30332 leader_election.cc:343] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2be86aaf16042299571a85097485a13 (127.14.212.67:37101): Not found: Tablet not found: f6a15be46af34060ac9be51d945b5302
I20251026 15:12:26.690798 30332 leader_election.cc:304] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3717f5812abb4c8cae8c7f1ca081ee69; no voters: 54d141fdf2874e50b643d9dafaf62410, d2be86aaf16042299571a85097485a13
I20251026 15:12:26.690907 30729 raft_consensus.cc:2749] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 15:12:27.212870 30730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:27.212610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:33941)
1026 15:12:27.212787 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:27.212824 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
I20251026 15:12:27.246647 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:27.246431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:52069)
1026 15:12:27.246565 (+ 134us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:27.246603 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20251026 15:12:27.374434 30732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:27.374154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:43089)
1026 15:12:27.374303 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:27.374352 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20251026 15:12:28.213634 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:28.213409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:51049)
1026 15:12:28.213551 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:28.213585 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20251026 15:12:28.247351 30734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:28.247101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:33989)
1026 15:12:28.247244 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:28.247277 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20251026 15:12:28.375151 30735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:28.374923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:39635)
1026 15:12:28.375068 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:28.375105 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20251026 15:12:29.146831 30736 raft_consensus.cc:493] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:12:29.146945 30736 raft_consensus.cc:515] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:29.147137 30736 leader_election.cc:290] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933), d2be86aaf16042299571a85097485a13 (127.14.212.67:37101)
W20251026 15:12:29.147542 30332 leader_election.cc:343] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933): Not found: Tablet not found: f6a15be46af34060ac9be51d945b5302
W20251026 15:12:29.147607 30332 leader_election.cc:343] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2be86aaf16042299571a85097485a13 (127.14.212.67:37101): Not found: Tablet not found: f6a15be46af34060ac9be51d945b5302
I20251026 15:12:29.147622 30332 leader_election.cc:304] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3717f5812abb4c8cae8c7f1ca081ee69; no voters: 54d141fdf2874e50b643d9dafaf62410, d2be86aaf16042299571a85097485a13
I20251026 15:12:29.147692 30736 raft_consensus.cc:2749] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 15:12:29.214625 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:29.214171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:57901)
1026 15:12:29.214375 (+ 204us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:29.214557 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20251026 15:12:29.248100 30738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:29.247863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:37601)
1026 15:12:29.248007 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:29.248053 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20251026 15:12:29.375974 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:29.375695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:44587)
1026 15:12:29.375871 (+ 176us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:29.375918 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":70,"threads_started":1}
I20251026 15:12:30.215552 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:30.215290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:33759)
1026 15:12:30.215439 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:30.215503 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20251026 15:12:30.248977 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:30.248731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:48103)
1026 15:12:30.248891 (+ 160us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:30.248929 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20251026 15:12:30.376755 30742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:30.376519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:56541)
1026 15:12:30.376660 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:30.376705 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20251026 15:12:31.216428 30743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:31.216120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:59213)
1026 15:12:31.216324 (+ 204us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:31.216379 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
I20251026 15:12:31.249779 30744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:31.249506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:56035)
1026 15:12:31.249676 (+ 170us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:31.249722 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20251026 15:12:31.377629 30745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:31.377323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:49365)
1026 15:12:31.377531 (+ 208us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:31.377580 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20251026 15:12:32.217293 30746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:32.217035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:46201)
1026 15:12:32.217197 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:32.217244 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20251026 15:12:32.250624 30747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:32.250337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:57505)
1026 15:12:32.250527 (+ 190us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:32.250575 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20251026 15:12:32.378516 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:32.378241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:36071)
1026 15:12:32.378379 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:32.378465 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20251026 15:12:33.218199 30749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:33.217912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:41953)
1026 15:12:33.218105 (+ 193us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:33.218150 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
I20251026 15:12:33.251448 30750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:33.251252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:59893)
1026 15:12:33.251373 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:33.251402 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20251026 15:12:33.379257 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:33.378997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:47421)
1026 15:12:33.379157 (+ 160us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:33.379203 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20251026 15:12:33.410980 30752 raft_consensus.cc:493] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:12:33.411087 30752 raft_consensus.cc:515] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d141fdf2874e50b643d9dafaf62410" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35933 } } peers { permanent_uuid: "d2be86aaf16042299571a85097485a13" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 37101 } } peers { permanent_uuid: "3717f5812abb4c8cae8c7f1ca081ee69" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 44075 } }
I20251026 15:12:33.411268 30752 leader_election.cc:290] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933), d2be86aaf16042299571a85097485a13 (127.14.212.67:37101)
W20251026 15:12:33.411743 30332 leader_election.cc:343] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2be86aaf16042299571a85097485a13 (127.14.212.67:37101): Not found: Tablet not found: f6a15be46af34060ac9be51d945b5302
W20251026 15:12:33.411818 30332 leader_election.cc:343] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 54d141fdf2874e50b643d9dafaf62410 (127.14.212.66:35933): Not found: Tablet not found: f6a15be46af34060ac9be51d945b5302
I20251026 15:12:33.411861 30332 leader_election.cc:304] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3717f5812abb4c8cae8c7f1ca081ee69; no voters: 54d141fdf2874e50b643d9dafaf62410, d2be86aaf16042299571a85097485a13
I20251026 15:12:33.411939 30752 raft_consensus.cc:2749] T f6a15be46af34060ac9be51d945b5302 P 3717f5812abb4c8cae8c7f1ca081ee69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 15:12:34.219064 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:34.218804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:41983)
1026 15:12:34.218968 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:34.219016 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":70,"threads_started":1}
I20251026 15:12:34.252370 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:34.252103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:38579)
1026 15:12:34.252277 (+ 174us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:34.252323 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":74,"threads_started":1}
I20251026 15:12:34.380175 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:34.379897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:52575)
1026 15:12:34.380061 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:34.380109 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
I20251026 15:12:35.220049 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:35.219733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:32785)
1026 15:12:35.219914 (+ 181us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:35.219984 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":67,"threads_started":1}
I20251026 15:12:35.253232 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:35.252940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:45591)
1026 15:12:35.253102 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:35.253144 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
I20251026 15:12:35.380900 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:35.380659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:49183)
1026 15:12:35.380815 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:35.380852 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
I20251026 15:12:36.220878 30759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:36.220659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:56145)
1026 15:12:36.220798 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:36.220831 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20251026 15:12:36.254050 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:36.253816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:49801)
1026 15:12:36.253964 (+ 148us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:36.254004 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20251026 15:12:36.381630 30761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:36.381399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:36287)
1026 15:12:36.381543 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:36.381582 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20251026 15:12:37.221657 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:37.221421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:36033)
1026 15:12:37.221568 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:37.221609 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20251026 15:12:37.254881 30763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:37.254620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:40691)
1026 15:12:37.254788 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:37.254834 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
I20251026 15:12:37.382499 30764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:37.382194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:58811)
1026 15:12:37.382359 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:37.382431 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20251026 15:12:38.222584 30765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:38.222265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.65:39681)
1026 15:12:38.222495 (+ 230us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:38.222538 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":75,"threads_started":1}
I20251026 15:12:38.255703 30766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:38.255486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.66:34695)
1026 15:12:38.255615 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:38.255657 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20251026 15:12:38.383294 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:38.383057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:32837 (local address 127.14.212.67:38487)
1026 15:12:38.383191 (+ 134us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:38.383238 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20251026 15:12:38.387331 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 30287
I20251026 15:12:38.392246 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 30451
I20251026 15:12:38.396770 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 30583
2025-10-26T15:12:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19194 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:12:38 dist-test-slave-8pgz krb5kdc[30770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:12:38 dist-test-slave-8pgz krb5kdc[30770](info): set up 2 sockets
Oct 26 15:12:38 dist-test-slave-8pgz krb5kdc[30770](info): commencing operation
krb5kdc: starting...
W20251026 15:12:40.440033 15185 mini_kdc.cc:121] Time spent starting KDC: real 2.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:12:40 dist-test-slave-8pgz krb5kdc[30770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:12:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:12:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:40.581638 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46453
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:46453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:40.665056 30786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:40.665270 30786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:40.665307 30786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:40.666826 30786 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20251026 15:12:40.666880 30786 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20251026 15:12:40.666900 30786 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20251026 15:12:40.666918 30786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:12:40.666941 30786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:40.666962 30786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:40.666983 30786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:12:40.667013 30786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:12:40.668788 30786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:46453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:40.669091 30786 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:40.669299 30786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:12:40.672564 30786 server_base.cc:1047] running on GCE node
W20251026 15:12:40.672658 30792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:40.672636 30791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:40.672680 30794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:40.673204 30786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:40.673509 30786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:40.674644 30786 hybrid_clock.cc:648] HybridClock initialized: now 1761491560674626 us; error 39 us; skew 500 ppm
Oct 26 15:12:40 dist-test-slave-8pgz krb5kdc[30770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491560, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:40.677215 30786 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:12:40.677793 30786 webserver.cc:492] Webserver started at http://127.14.212.126:38025/ using document root <none> and password file <none>
I20251026 15:12:40.677964 30786 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:40.677997 30786 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:40.678102 30786 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:12:40.679126 30786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "a1c8d85f9f844454a6abb5cd6994f3c6"
format_stamp: "Formatted at 2025-10-26 15:12:40 on dist-test-slave-8pgz"
server_key: "f856a81e1be3c1257713533f7ac88764"
server_key_iv: "fc6dad1a47d64b7c5ae28fbe8af55607"
server_key_version: "encryptionkey@0"
I20251026 15:12:40.679442 30786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "a1c8d85f9f844454a6abb5cd6994f3c6"
format_stamp: "Formatted at 2025-10-26 15:12:40 on dist-test-slave-8pgz"
server_key: "f856a81e1be3c1257713533f7ac88764"
server_key_iv: "fc6dad1a47d64b7c5ae28fbe8af55607"
server_key_version: "encryptionkey@0"
I20251026 15:12:40.680778 30786 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:12:40.681579 30801 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:40.681794 30786 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:12:40.681876 30786 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "a1c8d85f9f844454a6abb5cd6994f3c6"
format_stamp: "Formatted at 2025-10-26 15:12:40 on dist-test-slave-8pgz"
server_key: "f856a81e1be3c1257713533f7ac88764"
server_key_iv: "fc6dad1a47d64b7c5ae28fbe8af55607"
server_key_version: "encryptionkey@0"
I20251026 15:12:40.681937 30786 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:40.697136 30786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:40.704349 30786 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:40.704478 30786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:40.708403 30786 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:46453
I20251026 15:12:40.708498 30853 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:46453 every 8 connection(s)
I20251026 15:12:40.708781 30786 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:12:40.709431 30854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:40.712016 30854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Bootstrap starting.
I20251026 15:12:40.712635 30854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:40.712877 30854 log.cc:826] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:40.713709 30854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: No bootstrap required, opened a new log
I20251026 15:12:40.715057 30854 raft_consensus.cc:359] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } }
I20251026 15:12:40.715199 30854 raft_consensus.cc:385] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:40.715233 30854 raft_consensus.cc:740] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c8d85f9f844454a6abb5cd6994f3c6, State: Initialized, Role: FOLLOWER
I20251026 15:12:40.715358 30854 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } }
I20251026 15:12:40.715443 30854 raft_consensus.cc:399] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:12:40.715482 30854 raft_consensus.cc:493] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:12:40.715533 30854 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:40.716122 30854 raft_consensus.cc:515] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } }
I20251026 15:12:40.716236 30854 leader_election.cc:304] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1c8d85f9f844454a6abb5cd6994f3c6; no voters:
I20251026 15:12:40.716536 30854 leader_election.cc:290] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:12:40.716622 30857 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:12:40.716730 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 30786
I20251026 15:12:40.716814 30854 sys_catalog.cc:565] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:12:40.716841 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:12:40.716851 30857 raft_consensus.cc:697] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 1 LEADER]: Becoming Leader. State: Replica: a1c8d85f9f844454a6abb5cd6994f3c6, State: Running, Role: LEADER
I20251026 15:12:40.716972 30857 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } }
I20251026 15:12:40.717042 15185 external_mini_cluster.cc:1468] Setting key d27c823431c9eb0f5d39791550e2ad4e
I20251026 15:12:40.717298 30858 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } } }
I20251026 15:12:40.717361 30858 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:40.717497 30859 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1c8d85f9f844454a6abb5cd6994f3c6. Latest consensus state: current_term: 1 leader_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } } }
I20251026 15:12:40.717619 30859 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: This master's current role is: LEADER
I20251026 15:12:40.718216 30864 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:12:40.719120 30864 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:12:40.720728 30864 catalog_manager.cc:1357] Generated new cluster ID: 034052a8eff44103bf534483a55d0a2e
I20251026 15:12:40.720778 30864 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 26 15:12:40 dist-test-slave-8pgz krb5kdc[30770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:40.728215 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:40.717740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.14.212.126:46453)
1026 15:12:40.717894 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:40.717907 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:40.717936 (+ 29us) server_negotiation.cc:408] Connection header received
1026 15:12:40.718117 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:40.718124 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:40.718186 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:40.718345 (+ 159us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:40.719681 (+ 1336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:40.720872 (+ 1191us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:40.721692 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:40.722368 (+ 676us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:40.724568 (+ 2200us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:40.724575 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:40.724575 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:40.724603 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:40.726758 (+ 2155us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:40.727158 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:40.727160 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:40.727161 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:40.727220 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:40.727382 (+ 162us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:40.727383 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:40.727383 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:40.727617 (+ 234us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:40.727780 (+ 163us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:40.727892 (+ 112us) server_negotiation.cc:300] Negotiation successful
1026 15:12:40.728020 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20251026 15:12:40.728997 30864 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:12:40.729746 30864 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:12:40.746270 30864 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Generated new TSK 0
I20251026 15:12:40.746560 30864 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:40.802589 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46453
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:40.887629 30882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:40.887825 30882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:40.887866 30882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:40.889345 30882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:40.889400 30882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:40.889444 30882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:12:40.891165 30882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:40.891453 30882 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:40.891651 30882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:40.894353 30887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:40.894354 30890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:40.894426 30888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:40.894620 30882 server_base.cc:1047] running on GCE node
I20251026 15:12:40.894762 30882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:40.894932 30882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:40.896070 30882 hybrid_clock.cc:648] HybridClock initialized: now 1761491560896055 us; error 29 us; skew 500 ppm
Oct 26 15:12:40 dist-test-slave-8pgz krb5kdc[30770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491560, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:40.898731 30882 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:12:40.899255 30882 webserver.cc:492] Webserver started at http://127.14.212.65:36325/ using document root <none> and password file <none>
I20251026 15:12:40.899456 30882 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:40.899498 30882 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:40.899634 30882 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:12:40.900605 30882 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "91f5745f09a540a091c55eeb9b67f97b"
format_stamp: "Formatted at 2025-10-26 15:12:40 on dist-test-slave-8pgz"
server_key: "d339a5b928eb64999f9120ee33a18903"
server_key_iv: "f71d9e74b95517119afcf3915ea741b5"
server_key_version: "encryptionkey@0"
I20251026 15:12:40.900906 30882 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "91f5745f09a540a091c55eeb9b67f97b"
format_stamp: "Formatted at 2025-10-26 15:12:40 on dist-test-slave-8pgz"
server_key: "d339a5b928eb64999f9120ee33a18903"
server_key_iv: "f71d9e74b95517119afcf3915ea741b5"
server_key_version: "encryptionkey@0"
I20251026 15:12:40.902143 30882 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20251026 15:12:40.902943 30897 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:40.903133 30882 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251026 15:12:40.903211 30882 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "91f5745f09a540a091c55eeb9b67f97b"
format_stamp: "Formatted at 2025-10-26 15:12:40 on dist-test-slave-8pgz"
server_key: "d339a5b928eb64999f9120ee33a18903"
server_key_iv: "f71d9e74b95517119afcf3915ea741b5"
server_key_version: "encryptionkey@0"
I20251026 15:12:40.903280 30882 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:40.915532 30882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:40.917661 30882 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:40.917805 30882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:40.918036 30882 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:40.918373 30882 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:40.918491 30882 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:40.918550 30882 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:40.918578 30882 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:40.924489 30882 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:33189
I20251026 15:12:40.924562 31010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:33189 every 8 connection(s)
I20251026 15:12:40.924870 30882 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:12:40.927752 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 30882
I20251026 15:12:40.927865 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:12:40.928059 15185 external_mini_cluster.cc:1468] Setting key f9138f9302c14eb3b5bb0ac4198ba329
Oct 26 15:12:40 dist-test-slave-8pgz krb5kdc[30770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491560, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:40.935315 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:40.925648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:53517 (local address 127.14.212.126:46453)
1026 15:12:40.925714 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:40.925717 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:40.925871 (+ 154us) server_negotiation.cc:408] Connection header received
1026 15:12:40.926370 (+ 499us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:40.926371 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:40.926381 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:40.926456 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:40.927243 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:40.927746 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:40.928456 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:40.928610 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:40.931871 (+ 3261us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:40.931876 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:40.931877 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:40.931890 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:40.933526 (+ 1636us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:40.934130 (+ 604us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:40.934132 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:40.934132 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:40.934223 (+ 91us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:40.934518 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:40.934520 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:40.934520 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:40.934645 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:40.934714 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:40.935106 (+ 392us) server_negotiation.cc:300] Negotiation successful
1026 15:12:40.935200 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251026 15:12:40.935516 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:40.925597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:53517)
1026 15:12:40.925761 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:40.925793 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:40.926274 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:40.926527 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:40.926531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:40.926861 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:40.927154 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:40.927157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:40.927842 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:40.927843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:40.928375 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:40.928377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:40.928546 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:40.929868 (+ 1322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:40.929902 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:40.931737 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:40.933637 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:40.933647 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:40.933651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:40.934017 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:40.934308 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:40.934309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:40.934309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:40.934438 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:40.934716 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:40.934721 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:40.934920 (+ 199us) client_negotiation.cc:769] Sending connection context
1026 15:12:40.935058 (+ 138us) client_negotiation.cc:240] Negotiation successful
1026 15:12:40.935247 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20251026 15:12:40.936193 31011 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46453
I20251026 15:12:40.936344 31011 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:40.936563 31011 heartbeater.cc:507] Master 127.14.212.126:46453 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
I20251026 15:12:40.937096 30818 ts_manager.cc:194] Registered new tserver with Master: 91f5745f09a540a091c55eeb9b67f97b (127.14.212.65:33189)
I20251026 15:12:40.937918 30818 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:53517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:40.983144 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46453
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:41.066505 31018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:41.066804 31018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:41.066849 31018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:41.068890 31018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:41.068969 31018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:41.069016 31018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:12:41.070901 31018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:41.071168 31018 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:41.071341 31018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:41.074497 31023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:41.074497 31024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:41.074527 31018 server_base.cc:1047] running on GCE node
W20251026 15:12:41.074497 31026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:41.074939 31018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:41.075192 31018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:41.076390 31018 hybrid_clock.cc:648] HybridClock initialized: now 1761491561076371 us; error 29 us; skew 500 ppm
Oct 26 15:12:41 dist-test-slave-8pgz krb5kdc[30770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491561, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:41.079355 31018 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:12:41.079972 31018 webserver.cc:492] Webserver started at http://127.14.212.66:43191/ using document root <none> and password file <none>
I20251026 15:12:41.080178 31018 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:41.080225 31018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:41.080348 31018 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:12:41.081362 31018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "1f29ab1372c04d68bffbd82c2b408d3e"
format_stamp: "Formatted at 2025-10-26 15:12:41 on dist-test-slave-8pgz"
server_key: "016515d0092a49ebb2fff4b4ae8e5dd5"
server_key_iv: "c4d6b496d55c117799372024e4c5216a"
server_key_version: "encryptionkey@0"
I20251026 15:12:41.081693 31018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "1f29ab1372c04d68bffbd82c2b408d3e"
format_stamp: "Formatted at 2025-10-26 15:12:41 on dist-test-slave-8pgz"
server_key: "016515d0092a49ebb2fff4b4ae8e5dd5"
server_key_iv: "c4d6b496d55c117799372024e4c5216a"
server_key_version: "encryptionkey@0"
I20251026 15:12:41.083166 31018 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:12:41.084029 31033 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:41.084244 31018 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:12:41.084322 31018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "1f29ab1372c04d68bffbd82c2b408d3e"
format_stamp: "Formatted at 2025-10-26 15:12:41 on dist-test-slave-8pgz"
server_key: "016515d0092a49ebb2fff4b4ae8e5dd5"
server_key_iv: "c4d6b496d55c117799372024e4c5216a"
server_key_version: "encryptionkey@0"
I20251026 15:12:41.084391 31018 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:41.101446 31018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:41.103643 31018 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:41.103785 31018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:41.104017 31018 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:41.104353 31018 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:41.104389 31018 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:41.104424 31018 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:41.104450 31018 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:41.110599 31018 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:46399
I20251026 15:12:41.110658 31146 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:46399 every 8 connection(s)
I20251026 15:12:41.110960 31018 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
Oct 26 15:12:41 dist-test-slave-8pgz krb5kdc[30770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491561, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:41.118268 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 31018
I20251026 15:12:41.118381 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:12:41.118638 15185 external_mini_cluster.cc:1468] Setting key 2b4f3ffa230063c198d5de9e84a477ff
I20251026 15:12:41.120844 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.111693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:38415 (local address 127.14.212.126:46453)
1026 15:12:41.111747 (+ 54us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:41.111749 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:41.111899 (+ 150us) server_negotiation.cc:408] Connection header received
1026 15:12:41.112418 (+ 519us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:41.112419 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:41.112433 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:41.112493 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:41.113296 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.113806 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.114375 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.114566 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.117145 (+ 2579us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:41.117152 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:41.117153 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:41.117164 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:41.119361 (+ 2197us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:41.119819 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:41.119821 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:41.119821 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:41.119853 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:41.120116 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:41.120116 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:41.120117 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:41.120255 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:41.120332 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:41.120591 (+ 259us) server_negotiation.cc:300] Negotiation successful
1026 15:12:41.120712 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251026 15:12:41.120926 31149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.111652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:38415)
1026 15:12:41.111791 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:41.111824 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:41.112317 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:41.112543 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:41.112547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:41.112879 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:41.113164 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.113167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.113882 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.113882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:41.114290 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.114291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.114431 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.115451 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:41.115471 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:41.117014 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:41.119486 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:41.119495 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:41.119499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:41.119741 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:41.119959 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:41.119960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:41.119961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:41.120045 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:41.120319 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:41.120323 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:41.120463 (+ 140us) client_negotiation.cc:769] Sending connection context
1026 15:12:41.120557 (+ 94us) client_negotiation.cc:240] Negotiation successful
1026 15:12:41.120686 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20251026 15:12:41.121479 31147 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46453
I20251026 15:12:41.121592 31147 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:41.121860 31147 heartbeater.cc:507] Master 127.14.212.126:46453 requested a full tablet report, sending...
I20251026 15:12:41.122335 30818 ts_manager.cc:194] Registered new tserver with Master: 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66:46399)
I20251026 15:12:41.122763 30818 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:38415
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:12:41.170810 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46453
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:12:41.254038 31154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:12:41.254257 31154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:12:41.254285 31154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:12:41.255946 31154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:12:41.256009 31154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:12:41.256042 31154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:12:41.257802 31154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:12:41.258081 31154 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:12:41.258291 31154 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:12:41.261256 31162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:41.261261 31159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:12:41.261261 31160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:12:41.261530 31154 server_base.cc:1047] running on GCE node
I20251026 15:12:41.261703 31154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:12:41.261930 31154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:12:41.263070 31154 hybrid_clock.cc:648] HybridClock initialized: now 1761491561263018 us; error 67 us; skew 500 ppm
Oct 26 15:12:41 dist-test-slave-8pgz krb5kdc[30770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491561, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:12:41.265756 31154 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:12:41.266364 31154 webserver.cc:492] Webserver started at http://127.14.212.67:35707/ using document root <none> and password file <none>
I20251026 15:12:41.266587 31154 fs_manager.cc:362] Metadata directory not provided
I20251026 15:12:41.266625 31154 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:12:41.266712 31154 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:12:41.267653 31154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "3e77fed44451463e91f591155b5353da"
format_stamp: "Formatted at 2025-10-26 15:12:41 on dist-test-slave-8pgz"
server_key: "ad35c6a4592859bd5a8e884c5ee3a9cd"
server_key_iv: "bafd141f8a61cae65f1125d200812fb2"
server_key_version: "encryptionkey@0"
I20251026 15:12:41.267926 31154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "3e77fed44451463e91f591155b5353da"
format_stamp: "Formatted at 2025-10-26 15:12:41 on dist-test-slave-8pgz"
server_key: "ad35c6a4592859bd5a8e884c5ee3a9cd"
server_key_iv: "bafd141f8a61cae65f1125d200812fb2"
server_key_version: "encryptionkey@0"
I20251026 15:12:41.269161 31154 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:12:41.269917 31169 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:41.270139 31154 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:12:41.270217 31154 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "3e77fed44451463e91f591155b5353da"
format_stamp: "Formatted at 2025-10-26 15:12:41 on dist-test-slave-8pgz"
server_key: "ad35c6a4592859bd5a8e884c5ee3a9cd"
server_key_iv: "bafd141f8a61cae65f1125d200812fb2"
server_key_version: "encryptionkey@0"
I20251026 15:12:41.270277 31154 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:12:41.278807 31154 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:12:41.281306 31154 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:12:41.281433 31154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:12:41.281630 31154 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:12:41.281960 31154 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:12:41.281991 31154 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:41.282016 31154 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:12:41.282027 31154 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:12:41.287838 31154 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:33495
I20251026 15:12:41.287936 31282 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:33495 every 8 connection(s)
I20251026 15:12:41.288203 31154 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
Oct 26 15:12:41 dist-test-slave-8pgz krb5kdc[30770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491561, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:12:41.295810 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 31154
I20251026 15:12:41.295902 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:12:41.296077 15185 external_mini_cluster.cc:1468] Setting key 871fec8e7302739770a4a26674c983e7
I20251026 15:12:41.297729 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.289080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:33553 (local address 127.14.212.126:46453)
1026 15:12:41.289165 (+ 85us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:41.289167 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:41.289258 (+ 91us) server_negotiation.cc:408] Connection header received
1026 15:12:41.289761 (+ 503us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:41.289762 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:41.289773 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:41.289825 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:41.290680 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.291179 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.291794 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.291920 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.294528 (+ 2608us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:41.294534 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:41.294535 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:41.294543 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:41.296202 (+ 1659us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:41.296804 (+ 602us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:41.296804 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:41.296805 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:41.296835 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:41.297099 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:41.297100 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:41.297100 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:41.297207 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:41.297266 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:41.297576 (+ 310us) server_negotiation.cc:300] Negotiation successful
1026 15:12:41.297662 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251026 15:12:41.297878 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.288958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:33553)
1026 15:12:41.289113 (+ 155us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:41.289146 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:41.289679 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:41.289879 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:41.289882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:41.290226 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:12:41.290579 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.290583 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.291298 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.291299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:41.291712 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.291714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.291832 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.292857 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:12:41.292876 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:12:41.294374 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:12:41.296351 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:41.296361 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:41.296364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:41.296722 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:41.296930 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:12:41.296931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:12:41.296932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:12:41.297034 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:12:41.297290 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:12:41.297296 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:12:41.297435 (+ 139us) client_negotiation.cc:769] Sending connection context
1026 15:12:41.297544 (+ 109us) client_negotiation.cc:240] Negotiation successful
1026 15:12:41.297668 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20251026 15:12:41.298578 31283 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46453
I20251026 15:12:41.298673 31283 heartbeater.cc:461] Registering TS with master...
I20251026 15:12:41.298857 31283 heartbeater.cc:507] Master 127.14.212.126:46453 requested a full tablet report, sending...
I20251026 15:12:41.299249 30818 ts_manager.cc:194] Registered new tserver with Master: 3e77fed44451463e91f591155b5353da (127.14.212.67:33495)
I20251026 15:12:41.299655 30818 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:33553
I20251026 15:12:41.308056 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:12:41.315349 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.309486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.14.212.126:46453)
1026 15:12:41.309570 (+ 84us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:41.309573 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:41.309617 (+ 44us) server_negotiation.cc:408] Connection header received
1026 15:12:41.309690 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:41.309691 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:41.309703 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:41.309755 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:12:41.310548 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.311034 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.311623 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.311821 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.312827 (+ 1006us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:12:41.312833 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:12:41.312834 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:12:41.312840 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:12:41.314240 (+ 1400us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:41.314635 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:41.314637 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:41.314637 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:41.314662 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:12:41.314809 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:12:41.314809 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:12:41.314810 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:12:41.314935 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:12:41.315036 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:41.315149 (+ 113us) server_negotiation.cc:300] Negotiation successful
1026 15:12:41.315243 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251026 15:12:41.317097 30818 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:12:41.318594 30818 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:12:41.324502 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.320816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:33495 (local address 127.0.0.1:43452)
1026 15:12:41.321190 (+ 374us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:41.321200 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:41.321243 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:41.321521 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:41.321522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:41.321527 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:41.321773 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.321776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.323077 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.323078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:41.324022 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.324023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.324106 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.324148 (+ 42us) client_negotiation.cc:769] Sending connection context
1026 15:12:41.324203 (+ 55us) client_negotiation.cc:240] Negotiation successful
1026 15:12:41.324250 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":89,"threads_started":1}
I20251026 15:12:41.324610 31294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.320792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:33189 (local address 127.0.0.1:46876)
1026 15:12:41.321046 (+ 254us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:41.321077 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:41.321171 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:41.321518 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:41.321521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:41.321524 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:41.321837 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.321838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.323202 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.323203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:41.324021 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.324023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.324104 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.324148 (+ 44us) client_negotiation.cc:769] Sending connection context
1026 15:12:41.324203 (+ 55us) client_negotiation.cc:240] Negotiation successful
1026 15:12:41.324250 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":62,"threads_started":1}
I20251026 15:12:41.324888 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.320947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.14.212.67:33495)
1026 15:12:41.321366 (+ 419us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:41.321375 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:41.321384 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:12:41.321419 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:41.321421 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:41.321445 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:41.321508 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:41.321846 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.323000 (+ 1154us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.324113 (+ 1113us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.324678 (+ 565us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.324740 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:41.324769 (+ 29us) server_negotiation.cc:300] Negotiation successful
1026 15:12:41.324813 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":89,"threads_started":1}
I20251026 15:12:41.325158 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.321000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46399 (local address 127.0.0.1:40682)
1026 15:12:41.321283 (+ 283us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:41.321291 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:41.321334 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:41.321604 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:41.321605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:41.321606 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:41.321837 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.321838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.323936 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.323937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:41.324946 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.324949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.325017 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.325021 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:12:41.325052 (+ 31us) client_negotiation.cc:240] Negotiation successful
1026 15:12:41.325088 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":60,"threads_started":1}
I20251026 15:12:41.325559 31217 tablet_service.cc:1505] Processing CreateTablet for tablet b3bd2f37ecd14a5bb6ee066249ea6823 (DEFAULT_TABLE table=test-table [id=2e58c4e858284944ba4aeb3c138e7c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:12:41.325855 31217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3bd2f37ecd14a5bb6ee066249ea6823. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:41.325958 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.320880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.14.212.65:33189)
1026 15:12:41.321059 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:41.321068 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:41.321152 (+ 84us) server_negotiation.cc:408] Connection header received
1026 15:12:41.321308 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:41.321311 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:41.321330 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:41.321412 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:41.321906 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.323091 (+ 1185us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.325160 (+ 2069us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.325752 (+ 592us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.325805 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:41.325839 (+ 34us) server_negotiation.cc:300] Negotiation successful
1026 15:12:41.325880 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20251026 15:12:41.326066 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.321205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40682 (local address 127.14.212.66:46399)
1026 15:12:41.321375 (+ 170us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:41.321387 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:41.321396 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:12:41.321431 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:41.321433 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:41.321447 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:41.321525 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:41.323189 (+ 1664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.323850 (+ 661us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.325329 (+ 1479us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.325866 (+ 537us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.325904 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:41.325930 (+ 26us) server_negotiation.cc:300] Negotiation successful
1026 15:12:41.325975 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20251026 15:12:41.326508 30945 tablet_service.cc:1505] Processing CreateTablet for tablet b3bd2f37ecd14a5bb6ee066249ea6823 (DEFAULT_TABLE table=test-table [id=2e58c4e858284944ba4aeb3c138e7c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:12:41.326509 31081 tablet_service.cc:1505] Processing CreateTablet for tablet b3bd2f37ecd14a5bb6ee066249ea6823 (DEFAULT_TABLE table=test-table [id=2e58c4e858284944ba4aeb3c138e7c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:12:41.326778 30945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3bd2f37ecd14a5bb6ee066249ea6823. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:41.326778 31081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3bd2f37ecd14a5bb6ee066249ea6823. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:12:41.328444 31300 tablet_bootstrap.cc:492] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da: Bootstrap starting.
I20251026 15:12:41.329035 31300 tablet_bootstrap.cc:654] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:41.329312 31300 log.cc:826] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:41.329546 31302 tablet_bootstrap.cc:492] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Bootstrap starting.
I20251026 15:12:41.330111 31302 tablet_bootstrap.cc:654] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:41.330293 31300 tablet_bootstrap.cc:492] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da: No bootstrap required, opened a new log
I20251026 15:12:41.330363 31300 ts_tablet_manager.cc:1403] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:12:41.330446 31302 log.cc:826] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:41.330792 31301 tablet_bootstrap.cc:492] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Bootstrap starting.
I20251026 15:12:41.331118 31302 tablet_bootstrap.cc:492] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: No bootstrap required, opened a new log
I20251026 15:12:41.331183 31302 ts_tablet_manager.cc:1403] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 15:12:41.331797 31301 tablet_bootstrap.cc:654] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Neither blocks nor log segments found. Creating new log.
I20251026 15:12:41.332094 31301 log.cc:826] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Log is configured to *not* fsync() on all Append() calls
I20251026 15:12:41.332165 31300 raft_consensus.cc:359] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } }
I20251026 15:12:41.332332 31300 raft_consensus.cc:385] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:41.332377 31300 raft_consensus.cc:740] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e77fed44451463e91f591155b5353da, State: Initialized, Role: FOLLOWER
I20251026 15:12:41.332507 31300 consensus_queue.cc:260] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } }
I20251026 15:12:41.332711 31302 raft_consensus.cc:359] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } }
I20251026 15:12:41.332864 31302 raft_consensus.cc:385] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:41.332896 31302 raft_consensus.cc:740] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f29ab1372c04d68bffbd82c2b408d3e, State: Initialized, Role: FOLLOWER
I20251026 15:12:41.332913 31300 ts_tablet_manager.cc:1434] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:12:41.332955 31301 tablet_bootstrap.cc:492] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: No bootstrap required, opened a new log
I20251026 15:12:41.333029 31301 ts_tablet_manager.cc:1403] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:12:41.333038 31283 heartbeater.cc:499] Master 127.14.212.126:46453 was elected leader, sending a full tablet report...
I20251026 15:12:41.333021 31302 consensus_queue.cc:260] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } }
I20251026 15:12:41.333300 31302 ts_tablet_manager.cc:1434] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:12:41.333333 31147 heartbeater.cc:499] Master 127.14.212.126:46453 was elected leader, sending a full tablet report...
I20251026 15:12:41.334998 31301 raft_consensus.cc:359] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } }
I20251026 15:12:41.335161 31301 raft_consensus.cc:385] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:12:41.335186 31301 raft_consensus.cc:740] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f5745f09a540a091c55eeb9b67f97b, State: Initialized, Role: FOLLOWER
I20251026 15:12:41.335273 31301 consensus_queue.cc:260] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } }
I20251026 15:12:41.335533 31301 ts_tablet_manager.cc:1434] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251026 15:12:41.335572 31011 heartbeater.cc:499] Master 127.14.212.126:46453 was elected leader, sending a full tablet report...
W20251026 15:12:41.361529 31148 tablet.cc:2378] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 15:12:41.425683 31012 tablet.cc:2378] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:12:41.454509 31307 raft_consensus.cc:493] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:12:41.454651 31307 raft_consensus.cc:515] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } }
I20251026 15:12:41.455044 31307 leader_election.cc:290] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91f5745f09a540a091c55eeb9b67f97b (127.14.212.65:33189), 3e77fed44451463e91f591155b5353da (127.14.212.67:33495)
I20251026 15:12:41.457181 31149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.455231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:33189 (local address 127.14.212.66:58153)
1026 15:12:41.455273 (+ 42us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:41.455291 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:41.455344 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:41.455565 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:41.455567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:41.455569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:41.455746 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.455748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.456387 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.456388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:41.456957 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.456958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.457033 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.457037 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:12:41.457071 (+ 34us) client_negotiation.cc:240] Negotiation successful
1026 15:12:41.457109 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20251026 15:12:41.457584 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.455337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:58153 (local address 127.14.212.65:33189)
1026 15:12:41.455393 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:41.455395 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:41.455405 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:12:41.455440 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:41.455441 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:41.455452 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:41.455525 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:41.455828 (+ 303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.456298 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.457051 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.457485 (+ 434us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.457493 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:41.457513 (+ 20us) server_negotiation.cc:300] Negotiation successful
1026 15:12:41.457536 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251026 15:12:41.457870 30965 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3bd2f37ecd14a5bb6ee066249ea6823" candidate_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f5745f09a540a091c55eeb9b67f97b" is_pre_election: true
I20251026 15:12:41.458014 30965 raft_consensus.cc:2468] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f29ab1372c04d68bffbd82c2b408d3e in term 0.
I20251026 15:12:41.458034 31309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.455294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:33495 (local address 127.14.212.66:58711)
1026 15:12:41.455490 (+ 196us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:41.455501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:12:41.455584 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:12:41.455750 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:12:41.455751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:12:41.455752 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:12:41.455982 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.455984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.456665 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.456666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:12:41.457240 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:12:41.457244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.457839 (+ 595us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.457842 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:12:41.457900 (+ 58us) client_negotiation.cc:240] Negotiation successful
1026 15:12:41.457943 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":145,"threads_started":1}
I20251026 15:12:41.458045 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.455374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:58711 (local address 127.14.212.67:33495)
1026 15:12:41.455415 (+ 41us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:41.455434 (+ 19us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:41.455549 (+ 115us) server_negotiation.cc:408] Connection header received
1026 15:12:41.455670 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:41.455671 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:41.455683 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:41.455735 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:12:41.456059 (+ 324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.456557 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.457358 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.457805 (+ 447us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.457813 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:41.457923 (+ 110us) server_negotiation.cc:300] Negotiation successful
1026 15:12:41.457968 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251026 15:12:41.458289 31034 leader_election.cc:304] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f29ab1372c04d68bffbd82c2b408d3e, 91f5745f09a540a091c55eeb9b67f97b; no voters:
I20251026 15:12:41.458449 31307 raft_consensus.cc:2804] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:12:41.458485 31237 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3bd2f37ecd14a5bb6ee066249ea6823" candidate_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e77fed44451463e91f591155b5353da" is_pre_election: true
I20251026 15:12:41.458520 31307 raft_consensus.cc:493] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:12:41.458552 31307 raft_consensus.cc:3060] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:41.458608 31237 raft_consensus.cc:2468] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f29ab1372c04d68bffbd82c2b408d3e in term 0.
I20251026 15:12:41.459540 31307 raft_consensus.cc:515] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } }
I20251026 15:12:41.459725 31307 leader_election.cc:290] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [CANDIDATE]: Term 1 election: Requested vote from peers 91f5745f09a540a091c55eeb9b67f97b (127.14.212.65:33189), 3e77fed44451463e91f591155b5353da (127.14.212.67:33495)
I20251026 15:12:41.459895 30965 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3bd2f37ecd14a5bb6ee066249ea6823" candidate_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f5745f09a540a091c55eeb9b67f97b"
I20251026 15:12:41.459975 30965 raft_consensus.cc:3060] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:41.460016 31237 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3bd2f37ecd14a5bb6ee066249ea6823" candidate_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e77fed44451463e91f591155b5353da"
I20251026 15:12:41.460083 31237 raft_consensus.cc:3060] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:12:41.460733 31237 raft_consensus.cc:2468] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f29ab1372c04d68bffbd82c2b408d3e in term 1.
I20251026 15:12:41.460734 30965 raft_consensus.cc:2468] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f29ab1372c04d68bffbd82c2b408d3e in term 1.
I20251026 15:12:41.460951 31034 leader_election.cc:304] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f29ab1372c04d68bffbd82c2b408d3e, 91f5745f09a540a091c55eeb9b67f97b; no voters:
I20251026 15:12:41.461063 31307 raft_consensus.cc:2804] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:12:41.461205 31307 raft_consensus.cc:697] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 1 LEADER]: Becoming Leader. State: Replica: 1f29ab1372c04d68bffbd82c2b408d3e, State: Running, Role: LEADER
I20251026 15:12:41.461309 31307 consensus_queue.cc:237] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } }
I20251026 15:12:41.462036 30818 catalog_manager.cc:5649] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66). New cstate: current_term: 1 leader_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } health_report { overall_health: HEALTHY } } }
I20251026 15:12:41.474361 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.471895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.14.212.66:46399)
1026 15:12:41.471954 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1026 15:12:41.471956 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:12:41.471966 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:12:41.472012 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:12:41.472013 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:12:41.472030 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:12:41.472092 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:12:41.472404 (+ 312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.472934 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:12:41.473673 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:12:41.473847 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:12:41.473905 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:12:41.474022 (+ 117us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:12:41.474092 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1026 15:12:41.474229 (+ 137us) server_negotiation.cc:300] Negotiation successful
1026 15:12:41.474256 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251026 15:12:41.476239 31237 raft_consensus.cc:1275] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da [term 1 FOLLOWER]: Refusing update from remote peer 1f29ab1372c04d68bffbd82c2b408d3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:12:41.476291 30965 raft_consensus.cc:1275] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Refusing update from remote peer 1f29ab1372c04d68bffbd82c2b408d3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:12:41.476589 31310 consensus_queue.cc:1048] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:12:41.476711 31307 consensus_queue.cc:1048] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:12:41.479223 31313 mvcc.cc:204] Tried to move back new op lower bound from 7215069435804061696 to 7215069435746091008. Current Snapshot: MvccSnapshot[applied={T|T < 7215069435804061696}]
I20251026 15:12:41.479331 31315 mvcc.cc:204] Tried to move back new op lower bound from 7215069435804061696 to 7215069435746091008. Current Snapshot: MvccSnapshot[applied={T|T < 7215069435804061696}]
I20251026 15:12:41.481876 30818 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60234:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:12:41.481968 30818 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60234:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:12:41.482748 30818 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Sending DeleteTablet for 3 replicas of tablet b3bd2f37ecd14a5bb6ee066249ea6823
I20251026 15:12:41.483018 30945 tablet_service.cc:1552] Processing DeleteTablet for tablet b3bd2f37ecd14a5bb6ee066249ea6823 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:46876
I20251026 15:12:41.483121 31217 tablet_service.cc:1552] Processing DeleteTablet for tablet b3bd2f37ecd14a5bb6ee066249ea6823 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:43452
I20251026 15:12:41.483144 31081 tablet_service.cc:1552] Processing DeleteTablet for tablet b3bd2f37ecd14a5bb6ee066249ea6823 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:40682
I20251026 15:12:41.483366 31325 tablet_replica.cc:333] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: stopping tablet replica
I20251026 15:12:41.483368 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 30882
I20251026 15:12:41.483472 31323 tablet_replica.cc:333] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: stopping tablet replica
I20251026 15:12:41.483588 31323 raft_consensus.cc:2243] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:12:41.483702 31323 raft_consensus.cc:2272] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:12:41.485036 31324 tablet_replica.cc:333] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da: stopping tablet replica
I20251026 15:12:41.485136 31324 raft_consensus.cc:2243] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:12:41.485239 31324 raft_consensus.cc:2272] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:12:41.485612 31324 ts_tablet_manager.cc:1916] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:12:41.487138 31324 ts_tablet_manager.cc:1929] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:12:41.487248 31324 log.cc:1199] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/b3bd2f37ecd14a5bb6ee066249ea6823
I20251026 15:12:41.487534 31324 ts_tablet_manager.cc:1950] T b3bd2f37ecd14a5bb6ee066249ea6823 P 3e77fed44451463e91f591155b5353da: Deleting consensus metadata
I20251026 15:12:41.487937 30803 catalog_manager.cc:4985] TS 3e77fed44451463e91f591155b5353da (127.14.212.67:33495): tablet b3bd2f37ecd14a5bb6ee066249ea6823 (table test-table [id=2e58c4e858284944ba4aeb3c138e7c39]) successfully deleted
W20251026 15:12:41.490051 30802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:33189 (error 108)
I20251026 15:12:41.490238 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 31018
I20251026 15:12:41.490837 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.490504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:33189 (local address 127.0.0.1:46888)
1026 15:12:41.490759 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:41.490796 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
W20251026 15:12:41.491047 30802 catalog_manager.cc:4712] TS 91f5745f09a540a091c55eeb9b67f97b (127.14.212.65:33189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3bd2f37ecd14a5bb6ee066249ea6823: Network error: Client connection negotiation failed: client connection to 127.14.212.65:33189: connect: Connection refused (error 111)
I20251026 15:12:41.495666 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 31154
I20251026 15:12:41.495672 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:12:41.495517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46399 (local address 127.0.0.1:40694)
1026 15:12:41.495595 (+ 78us) negotiation.cc:107] Waiting for socket to connect
1026 15:12:41.495635 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.66:46399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20251026 15:12:41.500581 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 30786
I20251026 15:13:01.506680 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46453
--webserver_interface=127.14.212.126
--webserver_port=38025
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:46453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:01.586299 31327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:01.586519 31327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:01.586555 31327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:01.587954 31327 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20251026 15:13:01.587997 31327 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20251026 15:13:01.588017 31327 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20251026 15:13:01.588037 31327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:13:01.588060 31327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:01.588092 31327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:01.588119 31327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:13:01.588140 31327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:13:01.589738 31327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:46453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=38025
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:01.589969 31327 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:01.590152 31327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:01.595983 31333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:01.595983 31336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:01.595979 31334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:01.596006 31327 server_base.cc:1047] running on GCE node
I20251026 15:13:01.596505 31327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:01.596726 31327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:01.597858 31327 hybrid_clock.cc:648] HybridClock initialized: now 1761491581597842 us; error 36 us; skew 500 ppm
Oct 26 15:13:01 dist-test-slave-8pgz krb5kdc[30770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491581, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:01.600279 31327 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:13:01.600754 31327 webserver.cc:492] Webserver started at http://127.14.212.126:38025/ using document root <none> and password file <none>
I20251026 15:13:01.600944 31327 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:01.600996 31327 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:01.602241 31327 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:01.602864 31343 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:01.603025 31327 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:01.603092 31327 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "a1c8d85f9f844454a6abb5cd6994f3c6"
format_stamp: "Formatted at 2025-10-26 15:12:40 on dist-test-slave-8pgz"
server_key: "f856a81e1be3c1257713533f7ac88764"
server_key_iv: "fc6dad1a47d64b7c5ae28fbe8af55607"
server_key_version: "encryptionkey@0"
I20251026 15:13:01.603399 31327 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:01.619589 31327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:01.626482 31327 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:01.626664 31327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:01.631100 31327 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:46453
I20251026 15:13:01.631145 31395 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:46453 every 8 connection(s)
I20251026 15:13:01.631534 31327 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:13:01.632757 31396 sys_catalog.cc:263] Verifying existing consensus state
I20251026 15:13:01.633405 31396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Bootstrap starting.
I20251026 15:13:01.641633 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 31327
I20251026 15:13:01.641959 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:33189
--local_ip_for_outbound_sockets=127.14.212.65
--tserver_master_addrs=127.14.212.126:46453
--webserver_port=36325
--webserver_interface=127.14.212.65
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:13:01.643237 31396 log.cc:826] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:01.646092 31396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 15:13:01.646483 31396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Bootstrap complete.
I20251026 15:13:01.650136 31396 raft_consensus.cc:359] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } }
I20251026 15:13:01.650534 31396 raft_consensus.cc:740] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c8d85f9f844454a6abb5cd6994f3c6, State: Initialized, Role: FOLLOWER
I20251026 15:13:01.650703 31396 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } }
I20251026 15:13:01.650835 31396 raft_consensus.cc:399] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:13:01.650904 31396 raft_consensus.cc:493] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:13:01.650957 31396 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 1 FOLLOWER]: Advancing to term 2
I20251026 15:13:01.652338 31396 raft_consensus.cc:515] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } }
I20251026 15:13:01.652470 31396 leader_election.cc:304] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1c8d85f9f844454a6abb5cd6994f3c6; no voters:
I20251026 15:13:01.652639 31396 leader_election.cc:290] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251026 15:13:01.652709 31400 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 15:13:01.652907 31396 sys_catalog.cc:565] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:13:01.652920 31400 raft_consensus.cc:697] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 2 LEADER]: Becoming Leader. State: Replica: a1c8d85f9f844454a6abb5cd6994f3c6, State: Running, Role: LEADER
I20251026 15:13:01.653118 31400 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } }
I20251026 15:13:01.653470 31401 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } } }
I20251026 15:13:01.653548 31401 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:01.653801 31402 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1c8d85f9f844454a6abb5cd6994f3c6. Latest consensus state: current_term: 2 leader_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } } }
I20251026 15:13:01.653865 31402 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:01.653972 31407 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:13:01.654718 31407 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:13:01.654875 31407 catalog_manager.cc:1269] Loaded cluster ID: 034052a8eff44103bf534483a55d0a2e
I20251026 15:13:01.655095 31407 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:13:01.656410 31407 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:13:01.656589 31407 catalog_manager.cc:6033] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Loaded TSK: 0
I20251026 15:13:01.669730 31407 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Generated new TSK 1
I20251026 15:13:01.669862 31407 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251026 15:13:01.726832 31398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:01.727058 31398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:01.727104 31398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:01.728518 31398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:01.728561 31398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:01.728605 31398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:13:01.730230 31398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:33189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=36325
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:01.730496 31398 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:01.730682 31398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:01.733367 31425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:01.733352 31424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:01.733355 31427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:01.733843 31398 server_base.cc:1047] running on GCE node
I20251026 15:13:01.734011 31398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:01.734242 31398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:01.735447 31398 hybrid_clock.cc:648] HybridClock initialized: now 1761491581735440 us; error 35 us; skew 500 ppm
Oct 26 15:13:01 dist-test-slave-8pgz krb5kdc[30770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491581, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:01.738700 31398 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:13:01.739421 31398 webserver.cc:492] Webserver started at http://127.14.212.65:36325/ using document root <none> and password file <none>
I20251026 15:13:01.739660 31398 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:01.739730 31398 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:01.741515 31398 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:13:01.742277 31434 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:01.742508 31398 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:01.742575 31398 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "91f5745f09a540a091c55eeb9b67f97b"
format_stamp: "Formatted at 2025-10-26 15:12:40 on dist-test-slave-8pgz"
server_key: "d339a5b928eb64999f9120ee33a18903"
server_key_iv: "f71d9e74b95517119afcf3915ea741b5"
server_key_version: "encryptionkey@0"
I20251026 15:13:01.742848 31398 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:01.787782 31398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:01.789844 31398 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:01.789949 31398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:01.790177 31398 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:01.790650 31441 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251026 15:13:01.791762 31398 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 15:13:01.791806 31398 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 15:13:01.791844 31398 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251026 15:13:01.792382 31398 ts_tablet_manager.cc:616] Registered 1 tablets
I20251026 15:13:01.792415 31398 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251026 15:13:01.792488 31441 tablet_bootstrap.cc:492] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Bootstrap starting.
I20251026 15:13:01.800105 31398 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:33189
I20251026 15:13:01.800189 31548 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:33189 every 8 connection(s)
I20251026 15:13:01.800657 31398 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:13:01.804279 31441 log.cc:826] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:01.806696 31441 tablet_bootstrap.cc:492] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 15:13:01.807164 31441 tablet_bootstrap.cc:492] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Bootstrap complete.
I20251026 15:13:01.808087 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 31398
Oct 26 15:13:01 dist-test-slave-8pgz krb5kdc[30770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491581, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:01.808382 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:46399
--local_ip_for_outbound_sockets=127.14.212.66
--tserver_master_addrs=127.14.212.126:46453
--webserver_port=43191
--webserver_interface=127.14.212.66
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:13:01.808403 31441 ts_tablet_manager.cc:1403] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
I20251026 15:13:01.809675 31441 raft_consensus.cc:359] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } }
I20251026 15:13:01.810348 31441 raft_consensus.cc:740] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f5745f09a540a091c55eeb9b67f97b, State: Initialized, Role: FOLLOWER
I20251026 15:13:01.810592 31441 consensus_queue.cc:260] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } }
I20251026 15:13:01.810920 31441 ts_tablet_manager.cc:1434] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20251026 15:13:01.813123 31552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:01.801457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:36885 (local address 127.14.212.126:46453)
1026 15:13:01.801670 (+ 213us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:01.801682 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:01.802673 (+ 991us) server_negotiation.cc:408] Connection header received
1026 15:13:01.803394 (+ 721us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:01.803401 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:01.803461 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:01.803627 (+ 166us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:01.804615 (+ 988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.805898 (+ 1283us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:01.806522 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.806746 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:01.809036 (+ 2290us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:01.809061 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:01.809062 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:01.809085 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:01.811363 (+ 2278us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:01.811861 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:01.811864 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:01.811864 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:01.811932 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:01.812165 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:01.812166 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:01.812166 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:01.812397 (+ 231us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:01.812491 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:01.812825 (+ 334us) server_negotiation.cc:300] Negotiation successful
1026 15:13:01.812960 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":102,"threads_started":1}
I20251026 15:13:01.813169 31551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:01.801359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:36885)
1026 15:13:01.802515 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:01.802557 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:01.803170 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:01.803655 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:01.803659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:01.804082 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:01.804486 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:01.804490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.805996 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:01.805998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:01.806450 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:01.806452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.806568 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:01.807210 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:01.807239 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:01.808923 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:01.811451 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:01.811458 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:01.811461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:01.811753 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:01.812016 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:01.812016 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:01.812017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:01.812099 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:01.812466 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:01.812471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:01.812645 (+ 174us) client_negotiation.cc:769] Sending connection context
1026 15:13:01.812768 (+ 123us) client_negotiation.cc:240] Negotiation successful
1026 15:13:01.812932 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"thread_start_us":75,"threads_started":1}
I20251026 15:13:01.814224 31549 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46453
I20251026 15:13:01.814335 31549 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:01.814620 31549 heartbeater.cc:507] Master 127.14.212.126:46453 requested a full tablet report, sending...
I20251026 15:13:01.815351 31360 ts_manager.cc:194] Registered new tserver with Master: 91f5745f09a540a091c55eeb9b67f97b (127.14.212.65:33189)
I20251026 15:13:01.816609 31360 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:36885
I20251026 15:13:01.817649 31549 heartbeater.cc:499] Master 127.14.212.126:46453 was elected leader, sending a full tablet report...
I20251026 15:13:01.821378 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:01.815979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:33189 (local address 127.0.0.1:42396)
1026 15:13:01.818520 (+ 2541us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:01.818550 (+ 30us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:01.818635 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:01.818949 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:01.818951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:01.818954 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:01.819215 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:01.819218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.820610 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:01.820611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:01.821179 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:01.821181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.821241 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:01.821262 (+ 21us) client_negotiation.cc:769] Sending connection context
1026 15:13:01.821296 (+ 34us) client_negotiation.cc:240] Negotiation successful
1026 15:13:01.821323 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2503,"thread_start_us":92,"threads_started":1}
I20251026 15:13:01.822168 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:01.816224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42396 (local address 127.14.212.65:33189)
1026 15:13:01.818767 (+ 2543us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:01.818777 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:01.818787 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:13:01.818819 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:01.818821 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:01.818845 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:01.819294 (+ 449us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:01.819381 (+ 87us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.820520 (+ 1139us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:01.821277 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.821978 (+ 701us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:01.822030 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:01.822059 (+ 29us) server_negotiation.cc:300] Negotiation successful
1026 15:13:01.822095 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2506,"thread_start_us":76,"threads_started":1}
I20251026 15:13:01.822516 31483 tablet_service.cc:1552] Processing DeleteTablet for tablet b3bd2f37ecd14a5bb6ee066249ea6823 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:42396
I20251026 15:13:01.822997 31558 tablet_replica.cc:333] stopping tablet replica
I20251026 15:13:01.823117 31558 raft_consensus.cc:2243] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:01.823184 31558 raft_consensus.cc:2272] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:01.823602 31558 ts_tablet_manager.cc:1916] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:01.825560 31558 ts_tablet_manager.cc:1929] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:01.825626 31558 log.cc:1199] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal/wals/b3bd2f37ecd14a5bb6ee066249ea6823
I20251026 15:13:01.825877 31558 ts_tablet_manager.cc:1950] T b3bd2f37ecd14a5bb6ee066249ea6823 P 91f5745f09a540a091c55eeb9b67f97b: Deleting consensus metadata
I20251026 15:13:01.826357 31344 catalog_manager.cc:4985] TS 91f5745f09a540a091c55eeb9b67f97b (127.14.212.65:33189): tablet b3bd2f37ecd14a5bb6ee066249ea6823 (table test-table [id=2e58c4e858284944ba4aeb3c138e7c39]) successfully deleted
W20251026 15:13:01.892940 31554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:01.893148 31554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:01.893174 31554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:01.894680 31554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:01.894728 31554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:01.894754 31554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:13:01.896356 31554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:46399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=43191
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:01.896546 31554 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:01.896699 31554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:01.899542 31566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:01.899567 31564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:01.899554 31563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:01.900082 31554 server_base.cc:1047] running on GCE node
I20251026 15:13:01.900259 31554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:01.900491 31554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:01.901664 31554 hybrid_clock.cc:648] HybridClock initialized: now 1761491581901643 us; error 36 us; skew 500 ppm
Oct 26 15:13:01 dist-test-slave-8pgz krb5kdc[30770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491581, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:01.904633 31554 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:13:01.905288 31554 webserver.cc:492] Webserver started at http://127.14.212.66:43191/ using document root <none> and password file <none>
I20251026 15:13:01.905494 31554 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:01.905552 31554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:01.907107 31554 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:13:01.907866 31573 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:01.908097 31554 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:01.908182 31554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "1f29ab1372c04d68bffbd82c2b408d3e"
format_stamp: "Formatted at 2025-10-26 15:12:41 on dist-test-slave-8pgz"
server_key: "016515d0092a49ebb2fff4b4ae8e5dd5"
server_key_iv: "c4d6b496d55c117799372024e4c5216a"
server_key_version: "encryptionkey@0"
I20251026 15:13:01.908478 31554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:01.924934 31554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:01.927038 31554 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:01.927215 31554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:01.927587 31554 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:01.928120 31580 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251026 15:13:01.928954 31554 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 15:13:01.929006 31554 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 15:13:01.929046 31554 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251026 15:13:01.929603 31554 ts_tablet_manager.cc:616] Registered 1 tablets
I20251026 15:13:01.929638 31554 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251026 15:13:01.929728 31580 tablet_bootstrap.cc:492] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Bootstrap starting.
I20251026 15:13:01.935878 31554 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:46399
I20251026 15:13:01.936092 31687 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:46399 every 8 connection(s)
I20251026 15:13:01.936296 31554 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:13:01.941150 31580 log.cc:826] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Log is configured to *not* fsync() on all Append() calls
Oct 26 15:13:01 dist-test-slave-8pgz krb5kdc[30770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491581, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:01.943913 31580 tablet_bootstrap.cc:492] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 15:13:01.944191 31580 tablet_bootstrap.cc:492] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Bootstrap complete.
I20251026 15:13:01.944839 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 31554
I20251026 15:13:01.945181 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:33495
--local_ip_for_outbound_sockets=127.14.212.67
--tserver_master_addrs=127.14.212.126:46453
--webserver_port=35707
--webserver_interface=127.14.212.67
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:13:01.945518 31580 ts_tablet_manager.cc:1403] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20251026 15:13:01.946718 31580 raft_consensus.cc:359] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } }
I20251026 15:13:01.947052 31580 raft_consensus.cc:740] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f29ab1372c04d68bffbd82c2b408d3e, State: Initialized, Role: FOLLOWER
I20251026 15:13:01.947242 31580 consensus_queue.cc:260] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } } peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } }
I20251026 15:13:01.947335 31552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:01.937242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:46467 (local address 127.14.212.126:46453)
1026 15:13:01.937296 (+ 54us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:01.937299 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:01.937618 (+ 319us) server_negotiation.cc:408] Connection header received
1026 15:13:01.938452 (+ 834us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:01.938454 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:01.938472 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:01.938549 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:01.939845 (+ 1296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.940682 (+ 837us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:01.941446 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.941625 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:01.944067 (+ 2442us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:01.944071 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:01.944071 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:01.944081 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:01.945832 (+ 1751us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:01.946280 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:01.946281 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:01.946281 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:01.946309 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:01.946560 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:01.946561 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:01.946562 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:01.946695 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:01.947045 (+ 350us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:01.947101 (+ 56us) server_negotiation.cc:300] Negotiation successful
1026 15:13:01.947218 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20251026 15:13:01.947434 31690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:01.937114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:46467)
1026 15:13:01.937445 (+ 331us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:01.937497 (+ 52us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:01.938304 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:01.938655 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:01.938660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:01.939247 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:01.939724 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:01.939729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.940767 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:01.940768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:01.941353 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:01.941357 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.941487 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:01.942219 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:01.942244 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:01.943974 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:01.945905 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:01.945915 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:01.945918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:01.946201 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:01.946361 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:01.946361 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:01.946362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:01.946461 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:01.946774 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:01.946778 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:01.946935 (+ 157us) client_negotiation.cc:769] Sending connection context
1026 15:13:01.947036 (+ 101us) client_negotiation.cc:240] Negotiation successful
1026 15:13:01.947185 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":98,"threads_started":1}
I20251026 15:13:01.947892 31580 ts_tablet_manager.cc:1434] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251026 15:13:01.947942 31688 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46453
I20251026 15:13:01.948014 31688 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:01.948184 31688 heartbeater.cc:507] Master 127.14.212.126:46453 requested a full tablet report, sending...
I20251026 15:13:01.948678 31360 ts_manager.cc:194] Registered new tserver with Master: 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66:46399)
I20251026 15:13:01.949347 31360 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:46467
I20251026 15:13:01.950244 31688 heartbeater.cc:499] Master 127.14.212.126:46453 was elected leader, sending a full tablet report...
Oct 26 15:13:01 dist-test-slave-8pgz krb5kdc[30770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491581, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for kudu/127.14.212.66@KRBTEST.COM
I20251026 15:13:01.956743 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:01.949048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.14.212.66:46399)
1026 15:13:01.949250 (+ 202us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:01.949255 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:01.949262 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:13:01.949286 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:01.949287 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:01.949298 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:01.949355 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:01.949987 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.950904 (+ 917us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:01.951390 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.951599 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:01.953595 (+ 1996us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:01.953604 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:01.953604 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:01.953612 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:01.955490 (+ 1878us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:01.955972 (+ 482us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:01.955974 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:01.955975 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:01.956009 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:01.956220 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:01.956221 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:01.956221 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:01.956342 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:01.956426 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:01.956582 (+ 156us) server_negotiation.cc:300] Negotiation successful
1026 15:13:01.956673 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20251026 15:13:01.956777 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:01.948993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46399 (local address 127.0.0.1:51430)
1026 15:13:01.949046 (+ 53us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:01.949115 (+ 69us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:01.949159 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:01.949367 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:01.949368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:01.949775 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:01.949906 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:01.949908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.950972 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:01.950973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:01.951322 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:01.951323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:01.951387 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:01.951988 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:01.951992 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:01.953486 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:01.955589 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:01.955596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:01.955599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:01.955876 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:01.956079 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:01.956079 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:01.956080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:01.956115 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:01.956412 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:01.956418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:01.956477 (+ 59us) client_negotiation.cc:769] Sending connection context
1026 15:13:01.956557 (+ 80us) client_negotiation.cc:240] Negotiation successful
1026 15:13:01.956680 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20251026 15:13:01.957176 31622 tablet_service.cc:1552] Processing DeleteTablet for tablet b3bd2f37ecd14a5bb6ee066249ea6823 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:12:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:51430
I20251026 15:13:01.957851 31695 tablet_replica.cc:333] stopping tablet replica
I20251026 15:13:01.957999 31695 raft_consensus.cc:2243] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:01.958073 31695 raft_consensus.cc:2272] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:01.958454 31695 ts_tablet_manager.cc:1916] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:01.960408 31695 ts_tablet_manager.cc:1929] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:01.960474 31695 log.cc:1199] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/b3bd2f37ecd14a5bb6ee066249ea6823
I20251026 15:13:01.960695 31695 ts_tablet_manager.cc:1950] T b3bd2f37ecd14a5bb6ee066249ea6823 P 1f29ab1372c04d68bffbd82c2b408d3e: Deleting consensus metadata
I20251026 15:13:01.961154 31345 catalog_manager.cc:4985] TS 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66:46399): tablet b3bd2f37ecd14a5bb6ee066249ea6823 (table test-table [id=2e58c4e858284944ba4aeb3c138e7c39]) successfully deleted
W20251026 15:13:02.026935 31692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:02.027124 31692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:02.027148 31692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:02.028640 31692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:02.028685 31692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:02.028712 31692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:13:02.030331 31692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:33495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=35707
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:02.030576 31692 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:02.030763 31692 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:02.033277 31703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:02.033279 31701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:02.033453 31700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:02.033425 31692 server_base.cc:1047] running on GCE node
I20251026 15:13:02.033685 31692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:02.033891 31692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:02.035022 31692 hybrid_clock.cc:648] HybridClock initialized: now 1761491582035002 us; error 31 us; skew 500 ppm
Oct 26 15:13:02 dist-test-slave-8pgz krb5kdc[30770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491582, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:02.037509 31692 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:13:02.038010 31692 webserver.cc:492] Webserver started at http://127.14.212.67:35707/ using document root <none> and password file <none>
I20251026 15:13:02.038195 31692 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:02.038251 31692 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:02.039558 31692 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:02.040117 31710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:02.040284 31692 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:13:02.040359 31692 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "3e77fed44451463e91f591155b5353da"
format_stamp: "Formatted at 2025-10-26 15:12:41 on dist-test-slave-8pgz"
server_key: "ad35c6a4592859bd5a8e884c5ee3a9cd"
server_key_iv: "bafd141f8a61cae65f1125d200812fb2"
server_key_version: "encryptionkey@0"
I20251026 15:13:02.040627 31692 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:02.054956 31692 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:02.056725 31692 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:02.056861 31692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:02.057080 31692 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:02.057397 31692 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:02.057432 31692 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:02.057468 31692 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:02.057502 31692 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:02.063194 31692 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:33495
I20251026 15:13:02.063315 31823 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:33495 every 8 connection(s)
I20251026 15:13:02.063571 31692 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
Oct 26 15:13:02 dist-test-slave-8pgz krb5kdc[30770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491582, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:02.070574 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 31692
I20251026 15:13:02.072084 31552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.064317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:46069 (local address 127.14.212.126:46453)
1026 15:13:02.064378 (+ 61us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:02.064381 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:02.064537 (+ 156us) server_negotiation.cc:408] Connection header received
1026 15:13:02.065033 (+ 496us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:02.065034 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:02.065045 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:02.065099 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:02.065832 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.066313 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:02.066921 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.067032 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:02.069414 (+ 2382us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:02.069418 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:02.069419 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:02.069426 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:02.070851 (+ 1425us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:02.071273 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:02.071275 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:02.071275 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:02.071302 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:02.071483 (+ 181us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:02.071484 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:02.071484 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:02.071575 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:02.071637 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:02.071891 (+ 254us) server_negotiation.cc:300] Negotiation successful
1026 15:13:02.071995 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251026 15:13:02.072196 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.064247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:46069)
1026 15:13:02.064444 (+ 197us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:02.064471 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:02.064945 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:02.065162 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:02.065165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:02.065479 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:02.065759 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:02.065762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.066456 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:02.066457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:02.066844 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:02.066846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.066953 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:02.067933 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:02.067947 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:02.069305 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:02.070932 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:02.070939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:02.070942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:02.071196 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:02.071359 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:02.071360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:02.071361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:02.071428 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:02.071636 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:02.071642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:02.071771 (+ 129us) client_negotiation.cc:769] Sending connection context
1026 15:13:02.071867 (+ 96us) client_negotiation.cc:240] Negotiation successful
1026 15:13:02.071996 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
I20251026 15:13:02.072607 31824 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46453
I20251026 15:13:02.072705 31824 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:02.072916 31824 heartbeater.cc:507] Master 127.14.212.126:46453 requested a full tablet report, sending...
I20251026 15:13:02.073304 31360 ts_manager.cc:194] Registered new tserver with Master: 3e77fed44451463e91f591155b5353da (127.14.212.67:33495)
I20251026 15:13:02.073768 31360 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:46069
I20251026 15:13:02.076653 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.070973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.14.212.126:46453)
1026 15:13:02.071086 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:02.071088 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:02.071215 (+ 127us) server_negotiation.cc:408] Connection header received
1026 15:13:02.071285 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:02.071286 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:02.071297 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:02.071357 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:02.072068 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.072852 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:02.073531 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.073690 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:02.074603 (+ 913us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:02.074606 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:02.074606 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:02.074612 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:02.075818 (+ 1206us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:02.076085 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:02.076086 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:02.076086 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:02.076101 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:02.076210 (+ 109us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:02.076211 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:02.076211 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:02.076281 (+ 70us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:02.076421 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:02.076525 (+ 104us) server_negotiation.cc:300] Negotiation successful
1026 15:13:02.076590 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20251026 15:13:02.077698 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:13:02.080442 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.078044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.14.212.126:46453)
1026 15:13:02.078203 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:02.078205 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:02.078297 (+ 92us) server_negotiation.cc:408] Connection header received
1026 15:13:02.078381 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:02.078382 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:02.078531 (+ 149us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:02.078600 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:13:02.078875 (+ 275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.079384 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:02.079963 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.080131 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:02.080165 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:13:02.080209 (+ 44us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
1026 15:13:02.080371 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38026: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20251026 15:13:02.080502 31827 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38026: token expired
W20251026 15:13:02.080539 31286 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.212.126:46453)
I20251026 15:13:02.085677 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.080771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38036 (local address 127.14.212.126:46453)
1026 15:13:02.080822 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:02.080825 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:02.080832 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:13:02.080851 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:02.080852 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:02.080860 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:02.080907 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:02.081489 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.081914 (+ 425us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:02.082470 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.082645 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:02.083513 (+ 868us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:02.083516 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:02.083517 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:02.083523 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:02.084792 (+ 1269us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:02.085078 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:02.085079 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:02.085079 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:02.085102 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:02.085236 (+ 134us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:02.085237 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:02.085238 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:02.085317 (+ 79us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:02.085414 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:02.085503 (+ 89us) server_negotiation.cc:300] Negotiation successful
1026 15:13:02.085583 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251026 15:13:02.086714 31360 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:13:02.087963 31360 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:13:02.089814 31483 tablet_service.cc:1505] Processing CreateTablet for tablet 3ef157e62ff040d49ffb6299ef35fc66 (DEFAULT_TABLE table=test-table [id=7c698e215a584830845c9882cbf73239]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:02.090013 31483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef157e62ff040d49ffb6299ef35fc66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:02.091347 31441 tablet_bootstrap.cc:492] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b: Bootstrap starting.
I20251026 15:13:02.091539 31622 tablet_service.cc:1505] Processing CreateTablet for tablet 3ef157e62ff040d49ffb6299ef35fc66 (DEFAULT_TABLE table=test-table [id=7c698e215a584830845c9882cbf73239]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:02.091673 31622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef157e62ff040d49ffb6299ef35fc66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:02.091696 31441 tablet_bootstrap.cc:654] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:02.092308 31441 tablet_bootstrap.cc:492] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b: No bootstrap required, opened a new log
I20251026 15:13:02.092371 31441 ts_tablet_manager.cc:1403] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:02.092427 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.089632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:33495 (local address 127.0.0.1:54658)
1026 15:13:02.089677 (+ 45us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:02.089689 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:02.089728 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:02.090090 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:02.090091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:02.090093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:02.090243 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:02.090245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.091288 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:02.091289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:02.092180 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:02.092183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.092257 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:02.092261 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:02.092294 (+ 33us) client_negotiation.cc:240] Negotiation successful
1026 15:13:02.092355 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20251026 15:13:02.092495 31441 raft_consensus.cc:359] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:02.092568 31441 raft_consensus.cc:385] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:02.092604 31441 raft_consensus.cc:740] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f5745f09a540a091c55eeb9b67f97b, State: Initialized, Role: FOLLOWER
I20251026 15:13:02.092653 31441 consensus_queue.cc:260] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:02.092743 31441 ts_tablet_manager.cc:1434] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:02.093178 31580 tablet_bootstrap.cc:492] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e: Bootstrap starting.
I20251026 15:13:02.093405 31830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.089752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.14.212.67:33495)
1026 15:13:02.089927 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:02.089936 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:02.089946 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:13:02.089984 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:02.089986 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:02.090004 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:02.090068 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:02.090308 (+ 240us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.091209 (+ 901us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:02.092536 (+ 1327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.093195 (+ 659us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:02.093260 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:02.093291 (+ 31us) server_negotiation.cc:300] Negotiation successful
1026 15:13:02.093337 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20251026 15:13:02.093531 31580 tablet_bootstrap.cc:654] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:02.094058 31580 tablet_bootstrap.cc:492] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e: No bootstrap required, opened a new log
I20251026 15:13:02.094101 31580 ts_tablet_manager.cc:1403] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:02.094210 31580 raft_consensus.cc:359] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:02.094269 31580 raft_consensus.cc:385] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:02.094292 31580 raft_consensus.cc:740] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f29ab1372c04d68bffbd82c2b408d3e, State: Initialized, Role: FOLLOWER
I20251026 15:13:02.094344 31580 consensus_queue.cc:260] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:02.094316 31758 tablet_service.cc:1505] Processing CreateTablet for tablet 3ef157e62ff040d49ffb6299ef35fc66 (DEFAULT_TABLE table=test-table [id=7c698e215a584830845c9882cbf73239]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:02.094468 31580 ts_tablet_manager.cc:1434] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 15:13:02.094631 31758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef157e62ff040d49ffb6299ef35fc66. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:02.097077 31833 tablet_bootstrap.cc:492] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da: Bootstrap starting.
I20251026 15:13:02.097643 31833 tablet_bootstrap.cc:654] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:02.097893 31833 log.cc:826] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:02.098616 31833 tablet_bootstrap.cc:492] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da: No bootstrap required, opened a new log
I20251026 15:13:02.098671 31833 ts_tablet_manager.cc:1403] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:13:02.099756 31833 raft_consensus.cc:359] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:02.099874 31833 raft_consensus.cc:385] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:02.099895 31833 raft_consensus.cc:740] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e77fed44451463e91f591155b5353da, State: Initialized, Role: FOLLOWER
I20251026 15:13:02.099984 31833 consensus_queue.cc:260] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:02.100229 31833 ts_tablet_manager.cc:1434] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:13:02.100270 31824 heartbeater.cc:499] Master 127.14.212.126:46453 was elected leader, sending a full tablet report...
W20251026 15:13:02.187080 31689 tablet.cc:2378] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 15:13:02.303750 31550 tablet.cc:2378] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 15:13:02.314103 31825 tablet.cc:2378] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:13:02.336241 31693 raft_consensus.cc:493] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:13:02.336400 31693 raft_consensus.cc:515] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:02.336738 31693 leader_election.cc:290] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e77fed44451463e91f591155b5353da (127.14.212.67:33495), 91f5745f09a540a091c55eeb9b67f97b (127.14.212.65:33189)
I20251026 15:13:02.338997 31690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.336955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:33495 (local address 127.14.212.66:35119)
1026 15:13:02.337019 (+ 64us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:02.337033 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:02.337097 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:02.337310 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:02.337311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:02.337313 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:02.337482 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:02.337483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.338126 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:02.338127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:02.338725 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:02.338727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.338799 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:02.338846 (+ 47us) client_negotiation.cc:769] Sending connection context
1026 15:13:02.338882 (+ 36us) client_negotiation.cc:240] Negotiation successful
1026 15:13:02.338926 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20251026 15:13:02.339382 31830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.337045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:35119 (local address 127.14.212.67:33495)
1026 15:13:02.337178 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:02.337182 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:02.337192 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:13:02.337224 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:02.337225 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:02.337236 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:02.337294 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:02.337556 (+ 262us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.338049 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:02.338801 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.339276 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:02.339285 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:02.339308 (+ 23us) server_negotiation.cc:300] Negotiation successful
1026 15:13:02.339336 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251026 15:13:02.339602 31836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.336955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:33189 (local address 127.14.212.66:60253)
1026 15:13:02.337153 (+ 198us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:02.337168 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:02.337250 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:02.337441 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:02.337441 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:02.337443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:02.337617 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:02.337619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.338296 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:02.338297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:02.339353 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:02.339355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.339435 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:02.339439 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:02.339478 (+ 39us) client_negotiation.cc:240] Negotiation successful
1026 15:13:02.339519 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":18,"thread_start_us":102,"threads_started":1}
I20251026 15:13:02.339633 31778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ef157e62ff040d49ffb6299ef35fc66" candidate_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e77fed44451463e91f591155b5353da" is_pre_election: true
I20251026 15:13:02.339789 31778 raft_consensus.cc:2468] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f29ab1372c04d68bffbd82c2b408d3e in term 0.
I20251026 15:13:02.340013 31575 leader_election.cc:304] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f29ab1372c04d68bffbd82c2b408d3e, 3e77fed44451463e91f591155b5353da; no voters:
I20251026 15:13:02.340057 31837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.337092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:60253 (local address 127.14.212.65:33189)
1026 15:13:02.337224 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:02.337226 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:02.337236 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:13:02.337343 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:02.337344 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:02.337356 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:02.337423 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:02.337695 (+ 272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.338197 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:02.339442 (+ 1245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.339934 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:02.339943 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:02.339970 (+ 27us) server_negotiation.cc:300] Negotiation successful
1026 15:13:02.339994 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20251026 15:13:02.340157 31693 raft_consensus.cc:2804] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:13:02.340236 31693 raft_consensus.cc:493] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:13:02.340273 31693 raft_consensus.cc:3060] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:02.340276 31503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ef157e62ff040d49ffb6299ef35fc66" candidate_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f5745f09a540a091c55eeb9b67f97b" is_pre_election: true
I20251026 15:13:02.340435 31503 raft_consensus.cc:2468] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f29ab1372c04d68bffbd82c2b408d3e in term 0.
I20251026 15:13:02.341125 31693 raft_consensus.cc:515] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:02.341270 31693 leader_election.cc:290] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [CANDIDATE]: Term 1 election: Requested vote from peers 3e77fed44451463e91f591155b5353da (127.14.212.67:33495), 91f5745f09a540a091c55eeb9b67f97b (127.14.212.65:33189)
I20251026 15:13:02.341435 31778 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ef157e62ff040d49ffb6299ef35fc66" candidate_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e77fed44451463e91f591155b5353da"
I20251026 15:13:02.341449 31503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ef157e62ff040d49ffb6299ef35fc66" candidate_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f5745f09a540a091c55eeb9b67f97b"
I20251026 15:13:02.341506 31778 raft_consensus.cc:3060] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:02.341524 31503 raft_consensus.cc:3060] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:02.342217 31778 raft_consensus.cc:2468] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f29ab1372c04d68bffbd82c2b408d3e in term 1.
I20251026 15:13:02.342248 31503 raft_consensus.cc:2468] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f29ab1372c04d68bffbd82c2b408d3e in term 1.
I20251026 15:13:02.342376 31575 leader_election.cc:304] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f29ab1372c04d68bffbd82c2b408d3e, 3e77fed44451463e91f591155b5353da; no voters:
I20251026 15:13:02.342489 31693 raft_consensus.cc:2804] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:02.342626 31693 raft_consensus.cc:697] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [term 1 LEADER]: Becoming Leader. State: Replica: 1f29ab1372c04d68bffbd82c2b408d3e, State: Running, Role: LEADER
I20251026 15:13:02.342694 31693 consensus_queue.cc:237] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:02.343431 31360 catalog_manager.cc:5649] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66). New cstate: current_term: 1 leader_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } health_report { overall_health: UNKNOWN } } }
I20251026 15:13:02.375947 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.373814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.14.212.66:46399)
1026 15:13:02.373887 (+ 73us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:02.373889 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:02.373897 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:13:02.373941 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:02.373942 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:02.373958 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:02.374012 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:13:02.374282 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.374804 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:02.375403 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:02.375556 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:02.375590 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:13:02.375688 (+ 98us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:13:02.375759 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:02.375825 (+ 66us) server_negotiation.cc:300] Negotiation successful
1026 15:13:02.375862 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251026 15:13:02.377476 31778 raft_consensus.cc:1275] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da [term 1 FOLLOWER]: Refusing update from remote peer 1f29ab1372c04d68bffbd82c2b408d3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:02.377467 31503 raft_consensus.cc:1275] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Refusing update from remote peer 1f29ab1372c04d68bffbd82c2b408d3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:02.377820 31838 consensus_queue.cc:1048] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:02.377954 31693 consensus_queue.cc:1048] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:02.379935 31840 mvcc.cc:204] Tried to move back new op lower bound from 7215069521415360512 to 7215069521276162048. Current Snapshot: MvccSnapshot[applied={T|T < 7215069521415360512}]
I20251026 15:13:02.381167 31842 mvcc.cc:204] Tried to move back new op lower bound from 7215069521415360512 to 7215069521276162048. Current Snapshot: MvccSnapshot[applied={T|T < 7215069521415360512 or (T in {7215069521415360512})}]
I20251026 15:13:02.382171 31360 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38036:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:02.382256 31360 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38036:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:02.382989 31360 catalog_manager.cc:5936] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Sending DeleteTablet for 3 replicas of tablet 3ef157e62ff040d49ffb6299ef35fc66
I20251026 15:13:02.383265 31483 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef157e62ff040d49ffb6299ef35fc66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:02 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:42396
I20251026 15:13:02.383272 31622 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef157e62ff040d49ffb6299ef35fc66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:02 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:51430
I20251026 15:13:02.383251 31758 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef157e62ff040d49ffb6299ef35fc66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:02 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:54658
I20251026 15:13:02.383396 31695 tablet_replica.cc:333] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e: stopping tablet replica
I20251026 15:13:02.383453 31695 raft_consensus.cc:2243] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:13:02.383539 31695 raft_consensus.cc:2272] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:02.383581 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 31398
I20251026 15:13:02.383697 31851 tablet_replica.cc:333] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da: stopping tablet replica
I20251026 15:13:02.383806 31851 raft_consensus.cc:2243] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:02.383929 31851 raft_consensus.cc:2272] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:02.384490 31851 ts_tablet_manager.cc:1916] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:02.386190 31851 ts_tablet_manager.cc:1929] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:02.386261 31851 log.cc:1199] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/3ef157e62ff040d49ffb6299ef35fc66
I20251026 15:13:02.386569 31851 ts_tablet_manager.cc:1950] T 3ef157e62ff040d49ffb6299ef35fc66 P 3e77fed44451463e91f591155b5353da: Deleting consensus metadata
I20251026 15:13:02.386909 31345 catalog_manager.cc:4985] TS 3e77fed44451463e91f591155b5353da (127.14.212.67:33495): tablet 3ef157e62ff040d49ffb6299ef35fc66 (table test-table [id=7c698e215a584830845c9882cbf73239]) successfully deleted
W20251026 15:13:02.388459 31344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:33189 (error 108)
I20251026 15:13:02.388815 31695 ts_tablet_manager.cc:1916] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:02.388906 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.388788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:33189 (local address 127.0.0.1:42404)
1026 15:13:02.388834 (+ 46us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:02.388866 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20251026 15:13:02.389040 31344 catalog_manager.cc:4712] TS 91f5745f09a540a091c55eeb9b67f97b (127.14.212.65:33189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ef157e62ff040d49ffb6299ef35fc66: Network error: Client connection negotiation failed: client connection to 127.14.212.65:33189: connect: Connection refused (error 111)
I20251026 15:13:02.389940 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 31554
I20251026 15:13:02.395663 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 31692
I20251026 15:13:02.396576 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:02.395727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46399 (local address 127.0.0.1:51438)
1026 15:13:02.396509 (+ 782us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:02.396544 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.66:46399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":762}
I20251026 15:13:02.400009 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 31327
I20251026 15:13:22.407275 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46453
--webserver_interface=127.14.212.126
--webserver_port=38025
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:46453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:22.488447 31853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:22.488663 31853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:22.488700 31853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:22.490262 31853 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20251026 15:13:22.490320 31853 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20251026 15:13:22.490339 31853 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20251026 15:13:22.490362 31853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:13:22.490381 31853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:22.490439 31853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:22.490469 31853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:13:22.490497 31853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:13:22.492287 31853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:46453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=38025
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:22.492551 31853 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:22.492750 31853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:22.495831 31861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:22.495975 31859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:22.495844 31858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:22.496217 31853 server_base.cc:1047] running on GCE node
I20251026 15:13:22.496447 31853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:22.496701 31853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:22.497872 31853 hybrid_clock.cc:648] HybridClock initialized: now 1761491602497860 us; error 32 us; skew 500 ppm
Oct 26 15:13:22 dist-test-slave-8pgz krb5kdc[30770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491602, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:22.500685 31853 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:13:22.501300 31853 webserver.cc:492] Webserver started at http://127.14.212.126:38025/ using document root <none> and password file <none>
I20251026 15:13:22.501502 31853 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:22.501544 31853 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:22.503019 31853 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 15:13:22.503793 31868 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:22.504096 31853 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:22.504179 31853 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "a1c8d85f9f844454a6abb5cd6994f3c6"
format_stamp: "Formatted at 2025-10-26 15:12:40 on dist-test-slave-8pgz"
server_key: "f856a81e1be3c1257713533f7ac88764"
server_key_iv: "fc6dad1a47d64b7c5ae28fbe8af55607"
server_key_version: "encryptionkey@0"
I20251026 15:13:22.504477 31853 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:22.527695 31853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:22.533260 31853 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:22.533406 31853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:22.537612 31853 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:46453
I20251026 15:13:22.537653 31920 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:46453 every 8 connection(s)
I20251026 15:13:22.538019 31853 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:13:22.539258 31921 sys_catalog.cc:263] Verifying existing consensus state
I20251026 15:13:22.539791 31921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Bootstrap starting.
I20251026 15:13:22.542445 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 31853
I20251026 15:13:22.542809 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:33189
--local_ip_for_outbound_sockets=127.14.212.65
--tserver_master_addrs=127.14.212.126:46453
--webserver_port=36325
--webserver_interface=127.14.212.65
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:13:22.549957 31921 log.cc:826] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:22.552301 31921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 15:13:22.552593 31921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Bootstrap complete.
I20251026 15:13:22.554704 31921 raft_consensus.cc:359] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } }
I20251026 15:13:22.554898 31921 raft_consensus.cc:740] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c8d85f9f844454a6abb5cd6994f3c6, State: Initialized, Role: FOLLOWER
I20251026 15:13:22.555025 31921 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } }
I20251026 15:13:22.555101 31921 raft_consensus.cc:399] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:13:22.555142 31921 raft_consensus.cc:493] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:13:22.555202 31921 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 2 FOLLOWER]: Advancing to term 3
I20251026 15:13:22.556118 31921 raft_consensus.cc:515] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } }
I20251026 15:13:22.556212 31921 leader_election.cc:304] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1c8d85f9f844454a6abb5cd6994f3c6; no voters:
I20251026 15:13:22.556378 31921 leader_election.cc:290] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20251026 15:13:22.556430 31925 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 15:13:22.556576 31925 raft_consensus.cc:697] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [term 3 LEADER]: Becoming Leader. State: Replica: a1c8d85f9f844454a6abb5cd6994f3c6, State: Running, Role: LEADER
I20251026 15:13:22.556735 31925 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } }
I20251026 15:13:22.556629 31921 sys_catalog.cc:565] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:13:22.557065 31927 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1c8d85f9f844454a6abb5cd6994f3c6. Latest consensus state: current_term: 3 leader_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } } }
I20251026 15:13:22.557145 31927 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:22.557054 31926 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1c8d85f9f844454a6abb5cd6994f3c6" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46453 } } }
I20251026 15:13:22.557329 31926 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:22.557498 31929 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:13:22.558235 31929 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:13:22.558419 31929 catalog_manager.cc:1269] Loaded cluster ID: 034052a8eff44103bf534483a55d0a2e
I20251026 15:13:22.558609 31929 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:13:22.559988 31929 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:13:22.560163 31929 catalog_manager.cc:6033] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Loaded TSK: 0
I20251026 15:13:22.560200 31929 catalog_manager.cc:6033] T 00000000000000000000000000000000 P a1c8d85f9f844454a6abb5cd6994f3c6: Loaded TSK: 1
F20251026 15:13:22.560384 31929 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f18559d3dcd google::LogMessage::Fail() at ??:0
@ 0x7f18559d7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f18559d37cc google::LogMessage::Flush() at ??:0
@ 0x7f18559d4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f18542125b6 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f185420fdc3 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f185826167e kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f185826189a kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f18582619e3 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1858221707 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f185822c503 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1856ad53a5 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1856acb5d7 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1856e006db start_thread at ??:0
@ 0x7f1854f2871f clone at ??:0
W20251026 15:13:22.646351 31923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:22.646610 31923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:22.646651 31923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:22.648254 31923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:22.648308 31923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:22.648353 31923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:13:22.650101 31923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:33189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=36325
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:22.650358 31923 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:22.650574 31923 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:22.653610 31962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:22.653635 31963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:22.653836 31965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:22.653859 31923 server_base.cc:1047] running on GCE node
I20251026 15:13:22.654021 31923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:22.654207 31923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:22.655335 31923 hybrid_clock.cc:648] HybridClock initialized: now 1761491602655308 us; error 50 us; skew 500 ppm
Oct 26 15:13:22 dist-test-slave-8pgz krb5kdc[30770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491602, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:22.658164 31923 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:13:22.658836 31923 webserver.cc:492] Webserver started at http://127.14.212.65:36325/ using document root <none> and password file <none>
I20251026 15:13:22.659039 31923 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:22.659083 31923 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:22.660414 31923 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:22.661142 31972 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:22.661339 31923 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:22.661422 31923 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "91f5745f09a540a091c55eeb9b67f97b"
format_stamp: "Formatted at 2025-10-26 15:12:40 on dist-test-slave-8pgz"
server_key: "d339a5b928eb64999f9120ee33a18903"
server_key_iv: "f71d9e74b95517119afcf3915ea741b5"
server_key_version: "encryptionkey@0"
I20251026 15:13:22.661713 31923 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:22.673341 31923 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:22.675518 31923 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:22.675661 31923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:22.675889 31923 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:22.676347 31979 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251026 15:13:22.677558 31923 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 15:13:22.677611 31923 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 15:13:22.677649 31923 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251026 15:13:22.678232 31923 ts_tablet_manager.cc:616] Registered 1 tablets
I20251026 15:13:22.678265 31923 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251026 15:13:22.678364 31979 tablet_bootstrap.cc:492] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b: Bootstrap starting.
I20251026 15:13:22.686357 31923 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:33189
I20251026 15:13:22.686740 32086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:33189 every 8 connection(s)
I20251026 15:13:22.686908 31923 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:13:22.688012 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 31923
I20251026 15:13:22.688411 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:46399
--local_ip_for_outbound_sockets=127.14.212.66
--tserver_master_addrs=127.14.212.126:46453
--webserver_port=43191
--webserver_interface=127.14.212.66
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:13:22.691470 31979 log.cc:826] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:22.693605 31979 tablet_bootstrap.cc:492] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 15:13:22.693902 31979 tablet_bootstrap.cc:492] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b: Bootstrap complete.
I20251026 15:13:22.694869 31979 ts_tablet_manager.cc:1403] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.013s
I20251026 15:13:22.695719 31979 raft_consensus.cc:359] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:22.695914 31979 raft_consensus.cc:740] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f5745f09a540a091c55eeb9b67f97b, State: Initialized, Role: FOLLOWER
I20251026 15:13:22.696036 31979 consensus_queue.cc:260] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:22.696308 31979 ts_tablet_manager.cc:1434] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
W20251026 15:13:22.794468 32089 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:13:22.687634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:42923)
1026 15:13:22.687838 (+ 204us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:22.687879 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:22.689010 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:22.794233 (+105223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
W20251026 15:13:22.794728 32087 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:46453 (0 consecutive failures): Network error: Failed to ping master at 127.14.212.126:46453: Client connection negotiation failed: client connection to 127.14.212.126:46453: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251026 15:13:22.795099 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:22.794993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:48047)
1026 15:13:22.795042 (+ 49us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:22.795078 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20251026 15:13:22.795442 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:22.795365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:38825)
1026 15:13:22.795408 (+ 43us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:22.795429 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20251026 15:13:22.795542 32087 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251026 15:13:22.797784 32090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:22.797992 32090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:22.798032 32090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:22.799558 32090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:22.799603 32090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:22.799649 32090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:13:22.801352 32090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:46399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=43191
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:22.801597 32090 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:22.801790 32090 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:22.804570 32097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:22.804574 32098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:22.804574 32100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:22.804903 32090 server_base.cc:1047] running on GCE node
I20251026 15:13:22.805070 32090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:22.805289 32090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:22.806447 32090 hybrid_clock.cc:648] HybridClock initialized: now 1761491602806435 us; error 29 us; skew 500 ppm
Oct 26 15:13:22 dist-test-slave-8pgz krb5kdc[30770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491602, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:22.809108 32090 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:13:22.809654 32090 webserver.cc:492] Webserver started at http://127.14.212.66:43191/ using document root <none> and password file <none>
I20251026 15:13:22.809842 32090 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:22.809880 32090 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:22.811425 32090 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:22.812136 32107 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:22.812314 32090 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:22.812389 32090 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "1f29ab1372c04d68bffbd82c2b408d3e"
format_stamp: "Formatted at 2025-10-26 15:12:41 on dist-test-slave-8pgz"
server_key: "016515d0092a49ebb2fff4b4ae8e5dd5"
server_key_iv: "c4d6b496d55c117799372024e4c5216a"
server_key_version: "encryptionkey@0"
I20251026 15:13:22.812671 32090 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:22.834684 32090 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:22.836793 32090 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:22.836938 32090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:22.837164 32090 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:22.837582 32114 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20251026 15:13:22.838568 32114 ts_tablet_manager.cc:1863] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20251026 15:13:22.838843 32114 ts_tablet_manager.cc:1916] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:22.840903 32114 ts_tablet_manager.cc:1929] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:22.840984 32114 log.cc:1199] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/3ef157e62ff040d49ffb6299ef35fc66
I20251026 15:13:22.841533 32114 ts_tablet_manager.cc:1950] T 3ef157e62ff040d49ffb6299ef35fc66 P 1f29ab1372c04d68bffbd82c2b408d3e: Deleting consensus metadata
I20251026 15:13:22.841727 32090 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20251026 15:13:22.841787 32090 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20251026 15:13:22.841832 32090 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:22.841861 32090 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:22.847591 32090 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:46399
I20251026 15:13:22.847656 32221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:46399 every 8 connection(s)
I20251026 15:13:22.847940 32090 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:13:22.849184 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:22.848819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:59389)
1026 15:13:22.848960 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:22.849056 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
W20251026 15:13:22.849416 32222 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:46453 (0 consecutive failures): Network error: Failed to ping master at 127.14.212.126:46453: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
I20251026 15:13:22.849670 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:22.849591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:41621)
1026 15:13:22.849630 (+ 39us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:22.849654 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20251026 15:13:22.849915 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:22.849853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:46177)
1026 15:13:22.849884 (+ 31us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:22.849902 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20251026 15:13:22.850039 32222 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251026 15:13:22.856060 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 32090
I20251026 15:13:22.856346 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:33495
--local_ip_for_outbound_sockets=127.14.212.67
--tserver_master_addrs=127.14.212.126:46453
--webserver_port=35707
--webserver_interface=127.14.212.67
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:22.938184 32225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:22.938374 32225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:22.938437 32225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:22.939942 32225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:22.940004 32225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:22.940035 32225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:13:22.941800 32225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:33495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=35707
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:46453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:22.942016 32225 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:22.942170 32225 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:22.944862 32233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:22.944880 32230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:22.944970 32225 server_base.cc:1047] running on GCE node
W20251026 15:13:22.944919 32231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:22.945183 32225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:22.945377 32225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:22.946470 32225 hybrid_clock.cc:648] HybridClock initialized: now 1761491602946452 us; error 29 us; skew 500 ppm
Oct 26 15:13:22 dist-test-slave-8pgz krb5kdc[30770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491602, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:22.949031 32225 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:13:22.949609 32225 webserver.cc:492] Webserver started at http://127.14.212.67:35707/ using document root <none> and password file <none>
I20251026 15:13:22.949795 32225 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:22.949836 32225 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:22.951184 32225 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:22.951777 32240 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:22.951932 32225 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:13:22.952010 32225 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "3e77fed44451463e91f591155b5353da"
format_stamp: "Formatted at 2025-10-26 15:12:41 on dist-test-slave-8pgz"
server_key: "ad35c6a4592859bd5a8e884c5ee3a9cd"
server_key_iv: "bafd141f8a61cae65f1125d200812fb2"
server_key_version: "encryptionkey@0"
I20251026 15:13:22.952301 32225 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:22.972474 32225 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:22.974855 32225 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:22.975018 32225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:22.975279 32225 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:22.975593 32225 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:22.975626 32225 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:22.975664 32225 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:22.975697 32225 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:22.981807 32225 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:33495
I20251026 15:13:22.981896 32353 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:33495 every 8 connection(s)
I20251026 15:13:22.982285 32225 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:13:22.983397 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:22.983058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:58055)
1026 15:13:22.983193 (+ 135us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:22.983272 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
W20251026 15:13:22.983603 32354 heartbeater.cc:646] Failed to heartbeat to 127.14.212.126:46453 (0 consecutive failures): Network error: Failed to ping master at 127.14.212.126:46453: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
I20251026 15:13:22.983902 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:22.983821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:41981)
1026 15:13:22.983863 (+ 42us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:22.983887 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20251026 15:13:22.984190 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:22.984069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:37389)
1026 15:13:22.984135 (+ 66us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:22.984166 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20251026 15:13:22.984323 32354 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251026 15:13:22.991466 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 32225
I20251026 15:13:23.796262 32358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:23.795993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:37447)
1026 15:13:23.796165 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:23.796215 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":79,"threads_started":1}
I20251026 15:13:23.850596 32359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:23.850367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:40241)
1026 15:13:23.850520 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:23.850554 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20251026 15:13:23.984937 32360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:23.984682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:46747)
1026 15:13:23.984834 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:23.984880 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20251026 15:13:24.400226 32361 raft_consensus.cc:493] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:13:24.400384 32361 raft_consensus.cc:515] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:24.400709 32361 leader_election.cc:290] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e77fed44451463e91f591155b5353da (127.14.212.67:33495), 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66:46399)
Oct 26 15:13:24 dist-test-slave-8pgz krb5kdc[30770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491602, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.66@KRBTEST.COM
Oct 26 15:13:24 dist-test-slave-8pgz krb5kdc[30770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491602, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.67@KRBTEST.COM
I20251026 15:13:24.410087 32365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:24.401095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:33719 (local address 127.14.212.66:46399)
1026 15:13:24.401198 (+ 103us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:24.401207 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:24.401225 (+ 18us) server_negotiation.cc:408] Connection header received
1026 15:13:24.401570 (+ 345us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:24.401571 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:24.401607 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:24.401707 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:24.402667 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:24.403359 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:24.404056 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:24.404294 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:24.406365 (+ 2071us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:24.406373 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:24.406373 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:24.406418 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:24.408531 (+ 2113us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:24.409100 (+ 569us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:24.409102 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:24.409102 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:24.409148 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:24.409327 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:24.409328 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:24.409328 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:24.409527 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:24.409612 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:24.409865 (+ 253us) server_negotiation.cc:300] Negotiation successful
1026 15:13:24.409994 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":51,"threads_started":1}
I20251026 15:13:24.410074 32362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:24.400834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:33495 (local address 127.14.212.65:60395)
1026 15:13:24.400925 (+ 91us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:24.400935 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:24.400991 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:24.401785 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:24.401787 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:24.402240 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:24.402557 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:24.402562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:24.403928 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:24.403929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:24.404416 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:24.404418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:24.404487 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:24.404927 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:24.404931 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:24.406723 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:24.408671 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:24.408673 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:24.408673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:24.409003 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:24.409199 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:24.409200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:24.409200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:24.409271 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:24.409592 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:24.409597 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:24.409713 (+ 116us) client_negotiation.cc:769] Sending connection context
1026 15:13:24.409812 (+ 99us) client_negotiation.cc:240] Negotiation successful
1026 15:13:24.409966 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":51,"threads_started":1}
I20251026 15:13:24.410096 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:24.400991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:60395 (local address 127.14.212.67:33495)
1026 15:13:24.401162 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:24.401179 (+ 17us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:24.401201 (+ 22us) server_negotiation.cc:408] Connection header received
1026 15:13:24.401631 (+ 430us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:24.401632 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:24.401664 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:24.401780 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:24.402673 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:24.403819 (+ 1146us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:24.404494 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:24.404765 (+ 271us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:24.406812 (+ 2047us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:24.406819 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:24.406819 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:24.406838 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:24.408565 (+ 1727us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:24.409078 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:24.409082 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:24.409082 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:24.409139 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:24.409330 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:24.409331 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:24.409331 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:24.409526 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:24.409612 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:24.409868 (+ 256us) server_negotiation.cc:300] Negotiation successful
1026 15:13:24.409986 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20251026 15:13:24.410180 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:24.400987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:46399 (local address 127.14.212.65:33719)
1026 15:13:24.401084 (+ 97us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:24.401092 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:24.401149 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:24.401763 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:24.401766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:24.402239 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:24.402557 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:24.402562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:24.403445 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:24.403446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:24.403982 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:24.403984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:24.404073 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:24.404631 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:24.404658 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:24.406254 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:24.408618 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:24.408625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:24.408628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:24.409015 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:24.409211 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:24.409212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:24.409212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:24.409270 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:24.409593 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:24.409597 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:24.409713 (+ 116us) client_negotiation.cc:769] Sending connection context
1026 15:13:24.409812 (+ 99us) client_negotiation.cc:240] Negotiation successful
1026 15:13:24.410057 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":110,"thread_start_us":34,"threads_started":1}
W20251026 15:13:24.410835 31974 leader_election.cc:343] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e77fed44451463e91f591155b5353da (127.14.212.67:33495): Not found: Tablet not found: 3ef157e62ff040d49ffb6299ef35fc66
W20251026 15:13:24.411643 31974 leader_election.cc:343] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66:46399): Not found: Tablet not found: 3ef157e62ff040d49ffb6299ef35fc66
I20251026 15:13:24.411698 31974 leader_election.cc:304] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91f5745f09a540a091c55eeb9b67f97b; no voters: 1f29ab1372c04d68bffbd82c2b408d3e, 3e77fed44451463e91f591155b5353da
I20251026 15:13:24.411823 32361 raft_consensus.cc:2749] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 15:13:24.796960 32362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:24.796811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:44391)
1026 15:13:24.796882 (+ 71us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:24.796915 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20251026 15:13:24.851281 32366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:24.851070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:51125)
1026 15:13:24.851191 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:24.851236 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20251026 15:13:24.985636 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:24.985427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:46229)
1026 15:13:24.985552 (+ 125us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:24.985593 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20251026 15:13:25.797857 32368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:25.797599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:45283)
1026 15:13:25.797744 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:25.797789 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20251026 15:13:25.852017 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:25.851789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:44383)
1026 15:13:25.851929 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:25.851973 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20251026 15:13:25.986462 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:25.986185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:57889)
1026 15:13:25.986356 (+ 171us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:25.986415 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20251026 15:13:26.508653 32371 raft_consensus.cc:493] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:13:26.508761 32371 raft_consensus.cc:515] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:26.508929 32371 leader_election.cc:290] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e77fed44451463e91f591155b5353da (127.14.212.67:33495), 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66:46399)
W20251026 15:13:26.509404 31974 leader_election.cc:343] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66:46399): Not found: Tablet not found: 3ef157e62ff040d49ffb6299ef35fc66
W20251026 15:13:26.509495 31974 leader_election.cc:343] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e77fed44451463e91f591155b5353da (127.14.212.67:33495): Not found: Tablet not found: 3ef157e62ff040d49ffb6299ef35fc66
I20251026 15:13:26.509529 31974 leader_election.cc:304] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91f5745f09a540a091c55eeb9b67f97b; no voters: 1f29ab1372c04d68bffbd82c2b408d3e, 3e77fed44451463e91f591155b5353da
I20251026 15:13:26.509635 32371 raft_consensus.cc:2749] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 15:13:26.798620 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:26.798350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:37097)
1026 15:13:26.798516 (+ 166us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:26.798557 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
I20251026 15:13:26.852744 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:26.852519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:60015)
1026 15:13:26.852647 (+ 128us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:26.852690 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20251026 15:13:26.987247 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:26.987005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:50525)
1026 15:13:26.987141 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:26.987184 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20251026 15:13:27.799448 32375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:27.799226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:34233)
1026 15:13:27.799358 (+ 132us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:27.799401 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20251026 15:13:27.853637 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:27.853376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:33005)
1026 15:13:27.853511 (+ 135us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:27.853556 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20251026 15:13:27.987957 32377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:27.987701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:33773)
1026 15:13:27.987862 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:27.987906 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20251026 15:13:28.800189 32378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:28.799918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:54041)
1026 15:13:28.800100 (+ 182us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:28.800142 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20251026 15:13:28.854313 32379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:28.854117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:38109)
1026 15:13:28.854232 (+ 115us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:28.854266 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20251026 15:13:28.988821 32380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:28.988596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:54259)
1026 15:13:28.988732 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:28.988774 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20251026 15:13:29.588888 32381 raft_consensus.cc:493] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:13:29.589008 32381 raft_consensus.cc:515] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:29.589221 32381 leader_election.cc:290] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e77fed44451463e91f591155b5353da (127.14.212.67:33495), 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66:46399)
W20251026 15:13:29.589643 31974 leader_election.cc:343] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66:46399): Not found: Tablet not found: 3ef157e62ff040d49ffb6299ef35fc66
W20251026 15:13:29.589720 31974 leader_election.cc:343] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e77fed44451463e91f591155b5353da (127.14.212.67:33495): Not found: Tablet not found: 3ef157e62ff040d49ffb6299ef35fc66
I20251026 15:13:29.589735 31974 leader_election.cc:304] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91f5745f09a540a091c55eeb9b67f97b; no voters: 1f29ab1372c04d68bffbd82c2b408d3e, 3e77fed44451463e91f591155b5353da
I20251026 15:13:29.589807 32381 raft_consensus.cc:2749] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 15:13:29.800976 32382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:29.800722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:53305)
1026 15:13:29.800890 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:29.800930 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20251026 15:13:29.855106 32383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:29.854818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:58595)
1026 15:13:29.855000 (+ 182us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:29.855051 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20251026 15:13:29.989609 32384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:29.989378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:51805)
1026 15:13:29.989520 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:29.989560 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20251026 15:13:30.801733 32385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:30.801496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:45447)
1026 15:13:30.801630 (+ 134us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:30.801679 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20251026 15:13:30.855904 32386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:30.855650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:35441)
1026 15:13:30.855811 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:30.855855 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20251026 15:13:30.990437 32387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:30.990154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:33797)
1026 15:13:30.990307 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:30.990357 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20251026 15:13:31.802515 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:31.802222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:35273)
1026 15:13:31.802386 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:31.802466 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
I20251026 15:13:31.856606 32389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:31.856387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:48871)
1026 15:13:31.856511 (+ 124us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:31.856555 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20251026 15:13:31.991158 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:31.990922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:42315)
1026 15:13:31.991070 (+ 148us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:31.991108 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20251026 15:13:32.803288 32391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:32.803040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:48579)
1026 15:13:32.803195 (+ 155us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:32.803239 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20251026 15:13:32.857354 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:32.857118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:52261)
1026 15:13:32.857263 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:32.857307 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20251026 15:13:32.991921 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:32.991710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:40165)
1026 15:13:32.991832 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:32.991872 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20251026 15:13:33.193591 32394 raft_consensus.cc:493] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:13:33.193691 32394 raft_consensus.cc:515] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:33.193811 32394 leader_election.cc:290] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e77fed44451463e91f591155b5353da (127.14.212.67:33495), 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66:46399)
W20251026 15:13:33.194348 31974 leader_election.cc:343] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66:46399): Not found: Tablet not found: 3ef157e62ff040d49ffb6299ef35fc66
W20251026 15:13:33.194465 31974 leader_election.cc:343] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e77fed44451463e91f591155b5353da (127.14.212.67:33495): Not found: Tablet not found: 3ef157e62ff040d49ffb6299ef35fc66
I20251026 15:13:33.194497 31974 leader_election.cc:304] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91f5745f09a540a091c55eeb9b67f97b; no voters: 1f29ab1372c04d68bffbd82c2b408d3e, 3e77fed44451463e91f591155b5353da
I20251026 15:13:33.194588 32394 raft_consensus.cc:2749] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 15:13:33.804037 32395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:33.803816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:53877)
1026 15:13:33.803943 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:33.803990 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20251026 15:13:33.858172 32396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:33.857921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:37901)
1026 15:13:33.858078 (+ 157us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:33.858123 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":68,"threads_started":1}
I20251026 15:13:33.992672 32397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:33.992415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:53621)
1026 15:13:33.992583 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:33.992622 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20251026 15:13:34.804831 32398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:34.804609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:53437)
1026 15:13:34.804737 (+ 128us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:34.804781 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20251026 15:13:34.859073 32399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:34.858734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:35337)
1026 15:13:34.858957 (+ 223us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:34.859007 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":99,"threads_started":1}
I20251026 15:13:34.993441 32400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:34.993179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:35973)
1026 15:13:34.993348 (+ 169us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:34.993391 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20251026 15:13:35.805584 32401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:35.805352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:55521)
1026 15:13:35.805497 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:35.805535 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20251026 15:13:35.859812 32402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:35.859584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:46563)
1026 15:13:35.859717 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:35.859762 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20251026 15:13:35.994181 32403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:35.993963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:53727)
1026 15:13:35.994100 (+ 137us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:35.994132 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20251026 15:13:36.806316 32404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:36.806097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:53265)
1026 15:13:36.806233 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:36.806269 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20251026 15:13:36.860592 32405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:36.860332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:33215)
1026 15:13:36.860503 (+ 171us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:36.860543 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
I20251026 15:13:36.897547 32406 raft_consensus.cc:493] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:13:36.897661 32406 raft_consensus.cc:515] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e77fed44451463e91f591155b5353da" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 33495 } } peers { permanent_uuid: "1f29ab1372c04d68bffbd82c2b408d3e" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 46399 } } peers { permanent_uuid: "91f5745f09a540a091c55eeb9b67f97b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 33189 } }
I20251026 15:13:36.897853 32406 leader_election.cc:290] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e77fed44451463e91f591155b5353da (127.14.212.67:33495), 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66:46399)
W20251026 15:13:36.898334 31974 leader_election.cc:343] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f29ab1372c04d68bffbd82c2b408d3e (127.14.212.66:46399): Not found: Tablet not found: 3ef157e62ff040d49ffb6299ef35fc66
W20251026 15:13:36.898444 31974 leader_election.cc:343] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e77fed44451463e91f591155b5353da (127.14.212.67:33495): Not found: Tablet not found: 3ef157e62ff040d49ffb6299ef35fc66
I20251026 15:13:36.898469 31974 leader_election.cc:304] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91f5745f09a540a091c55eeb9b67f97b; no voters: 1f29ab1372c04d68bffbd82c2b408d3e, 3e77fed44451463e91f591155b5353da
I20251026 15:13:36.898552 32406 raft_consensus.cc:2749] T 3ef157e62ff040d49ffb6299ef35fc66 P 91f5745f09a540a091c55eeb9b67f97b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 15:13:36.994928 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:36.994694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:49831)
1026 15:13:36.994828 (+ 134us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:36.994875 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20251026 15:13:37.807121 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:37.806876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.65:33633)
1026 15:13:37.807032 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:37.807075 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
I20251026 15:13:37.861483 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:37.861234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.66:37179)
1026 15:13:37.861387 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:37.861434 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
I20251026 15:13:37.995780 32410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:37.995537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46453 (local address 127.14.212.67:53149)
1026 15:13:37.995681 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:37.995727 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.126:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20251026 15:13:38.001338 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 31923
I20251026 15:13:38.006585 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 32090
I20251026 15:13:38.011188 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 32225
2025-10-26T15:13:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59613 ms)
[----------] 31 tests from SecurityITest (365717 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:13:38 dist-test-slave-8pgz krb5kdc[32413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:13:38 dist-test-slave-8pgz krb5kdc[32413](info): set up 2 sockets
Oct 26 15:13:38 dist-test-slave-8pgz krb5kdc[32413](info): commencing operation
krb5kdc: starting...
W20251026 15:13:40.035748 15185 mini_kdc.cc:121] Time spent starting KDC: real 2.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:13:40 dist-test-slave-8pgz krb5kdc[32413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:13:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:13:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:40.167399 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:43755
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:43755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:40.246848 32429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:40.247040 32429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:40.247062 32429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:40.248564 32429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:13:40.248616 32429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:40.248631 32429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:40.248643 32429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:13:40.248656 32429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:13:40.250344 32429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:43755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:43755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:40.250614 32429 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:40.250808 32429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:40.253708 32435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:40.253784 32434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:40.253726 32437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:40.253805 32429 server_base.cc:1047] running on GCE node
I20251026 15:13:40.254187 32429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:40.254419 32429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:40.255578 32429 hybrid_clock.cc:648] HybridClock initialized: now 1761491620255538 us; error 59 us; skew 500 ppm
Oct 26 15:13:40 dist-test-slave-8pgz krb5kdc[32413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491620, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:40.258069 32429 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:13:40.258630 32429 webserver.cc:492] Webserver started at http://127.14.212.126:37951/ using document root <none> and password file <none>
I20251026 15:13:40.258818 32429 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:40.258855 32429 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:40.258975 32429 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:40.259927 32429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "70fb1e153c5f4011ad49c5071b02acda"
format_stamp: "Formatted at 2025-10-26 15:13:40 on dist-test-slave-8pgz"
server_key: "caf6fb47b93eafc4bfef88853b0b887d"
server_key_iv: "da9894b2337a13f2e34b56107566dff6"
server_key_version: "encryptionkey@0"
I20251026 15:13:40.260246 32429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "70fb1e153c5f4011ad49c5071b02acda"
format_stamp: "Formatted at 2025-10-26 15:13:40 on dist-test-slave-8pgz"
server_key: "caf6fb47b93eafc4bfef88853b0b887d"
server_key_iv: "da9894b2337a13f2e34b56107566dff6"
server_key_version: "encryptionkey@0"
I20251026 15:13:40.261438 32429 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:13:40.262254 32444 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:40.262482 32429 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:40.262569 32429 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "70fb1e153c5f4011ad49c5071b02acda"
format_stamp: "Formatted at 2025-10-26 15:13:40 on dist-test-slave-8pgz"
server_key: "caf6fb47b93eafc4bfef88853b0b887d"
server_key_iv: "da9894b2337a13f2e34b56107566dff6"
server_key_version: "encryptionkey@0"
I20251026 15:13:40.262632 32429 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:40.274381 32429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:40.280833 32429 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:40.281031 32429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:40.285061 32429 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:43755
I20251026 15:13:40.285101 32496 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:43755 every 8 connection(s)
I20251026 15:13:40.285411 32429 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:13:40.285981 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:40.288511 32497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda: Bootstrap starting.
I20251026 15:13:40.289104 32497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:40.289374 32497 log.cc:826] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:40.290160 32497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda: No bootstrap required, opened a new log
I20251026 15:13:40.291431 32497 raft_consensus.cc:359] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fb1e153c5f4011ad49c5071b02acda" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43755 } }
I20251026 15:13:40.291566 32497 raft_consensus.cc:385] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:40.291587 32497 raft_consensus.cc:740] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70fb1e153c5f4011ad49c5071b02acda, State: Initialized, Role: FOLLOWER
I20251026 15:13:40.291702 32497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fb1e153c5f4011ad49c5071b02acda" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43755 } }
I20251026 15:13:40.291777 32497 raft_consensus.cc:399] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:13:40.291801 32497 raft_consensus.cc:493] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:13:40.291837 32497 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:40.292034 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 32429
I20251026 15:13:40.292143 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:13:40.292339 15185 external_mini_cluster.cc:1468] Setting key e0dcd16d931485ee95c5a2af1121a257
I20251026 15:13:40.292613 32497 raft_consensus.cc:515] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fb1e153c5f4011ad49c5071b02acda" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43755 } }
I20251026 15:13:40.292721 32497 leader_election.cc:304] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70fb1e153c5f4011ad49c5071b02acda; no voters:
I20251026 15:13:40.292923 32497 leader_election.cc:290] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:13:40.293020 32501 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:40.293160 32501 raft_consensus.cc:697] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [term 1 LEADER]: Becoming Leader. State: Replica: 70fb1e153c5f4011ad49c5071b02acda, State: Running, Role: LEADER
I20251026 15:13:40.293228 32497 sys_catalog.cc:565] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:13:40.293264 32501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fb1e153c5f4011ad49c5071b02acda" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43755 } }
I20251026 15:13:40.294680 32503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70fb1e153c5f4011ad49c5071b02acda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fb1e153c5f4011ad49c5071b02acda" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43755 } } }
I20251026 15:13:40.294950 32503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:40.294967 32504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70fb1e153c5f4011ad49c5071b02acda. Latest consensus state: current_term: 1 leader_uuid: "70fb1e153c5f4011ad49c5071b02acda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fb1e153c5f4011ad49c5071b02acda" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43755 } } }
I20251026 15:13:40.295316 32504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda [sys.catalog]: This master's current role is: LEADER
W20251026 15:13:40.295727 32516 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:13:40.295780 32516 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:13:40.296384 32517 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:13:40.296590 32517 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Oct 26 15:13:40 dist-test-slave-8pgz krb5kdc[32413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:40.298893 32517 catalog_manager.cc:1357] Generated new cluster ID: fa4f73dd738646a989a493bc5adcf458
I20251026 15:13:40.298949 32517 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:13:40.302264 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.292836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.14.212.126:43755)
1026 15:13:40.293038 (+ 202us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:40.293051 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:40.293079 (+ 28us) server_negotiation.cc:408] Connection header received
1026 15:13:40.293192 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:40.293197 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:40.293249 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:40.293393 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:40.294144 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.295419 (+ 1275us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.296162 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.296427 (+ 265us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.298807 (+ 2380us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:40.298832 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:40.298832 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:40.298856 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:40.300884 (+ 2028us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:40.301285 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:40.301287 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:40.301287 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:40.301342 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:40.301494 (+ 152us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:40.301495 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:40.301495 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:40.301724 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:40.301854 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:40.301975 (+ 121us) server_negotiation.cc:300] Negotiation successful
1026 15:13:40.302092 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":65,"threads_started":1}
I20251026 15:13:40.315765 32517 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:13:40.316571 32517 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:13:40.329895 32517 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda: Generated new TSK 0
I20251026 15:13:40.330173 32517 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:40.390966 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:43755
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:40.473485 32525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:40.473685 32525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:40.473711 32525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:40.475248 32525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:40.475294 32525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:40.475322 32525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:13:40.476922 32525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:43755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:40.477133 32525 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:40.477329 32525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:40.480002 32533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:40.480049 32531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:40.480010 32530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:40.480067 32525 server_base.cc:1047] running on GCE node
I20251026 15:13:40.480381 32525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:40.480576 32525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:40.481724 32525 hybrid_clock.cc:648] HybridClock initialized: now 1761491620481714 us; error 31 us; skew 500 ppm
Oct 26 15:13:40 dist-test-slave-8pgz krb5kdc[32413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491620, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:40.484314 32525 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:13:40.484845 32525 webserver.cc:492] Webserver started at http://127.14.212.65:35665/ using document root <none> and password file <none>
I20251026 15:13:40.485086 32525 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:40.485127 32525 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:40.485253 32525 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:40.486217 32525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "5603764075d641dc9468cbd0a7b9f79d"
format_stamp: "Formatted at 2025-10-26 15:13:40 on dist-test-slave-8pgz"
server_key: "070a72b7cc6f458574943180dfbbdfae"
server_key_iv: "a01a7abfaa73f06c86e39ced9a74c6ea"
server_key_version: "encryptionkey@0"
I20251026 15:13:40.486569 32525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "5603764075d641dc9468cbd0a7b9f79d"
format_stamp: "Formatted at 2025-10-26 15:13:40 on dist-test-slave-8pgz"
server_key: "070a72b7cc6f458574943180dfbbdfae"
server_key_iv: "a01a7abfaa73f06c86e39ced9a74c6ea"
server_key_version: "encryptionkey@0"
I20251026 15:13:40.487800 32525 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:40.488587 32540 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:40.488780 32525 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:40.488857 32525 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "5603764075d641dc9468cbd0a7b9f79d"
format_stamp: "Formatted at 2025-10-26 15:13:40 on dist-test-slave-8pgz"
server_key: "070a72b7cc6f458574943180dfbbdfae"
server_key_iv: "a01a7abfaa73f06c86e39ced9a74c6ea"
server_key_version: "encryptionkey@0"
I20251026 15:13:40.488956 32525 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:40.502882 32525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:40.504773 32525 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:40.504910 32525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:40.505162 32525 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:40.505471 32525 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:40.505503 32525 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:40.505541 32525 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:40.505568 32525 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:40.511135 32525 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:45575
I20251026 15:13:40.511366 32653 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:45575 every 8 connection(s)
I20251026 15:13:40.511511 32525 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:13:40.515843 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 32525
I20251026 15:13:40.515995 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:13:40.516222 15185 external_mini_cluster.cc:1468] Setting key 2d20589de6456faf5ebe1baaf591f584
Oct 26 15:13:40 dist-test-slave-8pgz krb5kdc[32413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491620, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:40.521095 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.512251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:42709 (local address 127.14.212.126:43755)
1026 15:13:40.512330 (+ 79us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:40.512334 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:40.512432 (+ 98us) server_negotiation.cc:408] Connection header received
1026 15:13:40.512946 (+ 514us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:40.512947 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:40.512964 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:40.513043 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:40.513824 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.514594 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.515219 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.515399 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.518244 (+ 2845us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:40.518248 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:40.518249 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:40.518257 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:40.519773 (+ 1516us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:40.520133 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:40.520134 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:40.520135 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:40.520167 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:40.520407 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:40.520408 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:40.520408 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:40.520505 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:40.520568 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:40.520942 (+ 374us) server_negotiation.cc:300] Negotiation successful
1026 15:13:40.521029 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251026 15:13:40.521370 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.512150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43755 (local address 127.14.212.65:42709)
1026 15:13:40.512315 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:40.512351 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:40.512850 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:40.513111 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:40.513114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:40.513447 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:40.513722 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.513726 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.514675 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.514676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:40.515133 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.515135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.515275 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.516417 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:40.516441 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:40.518109 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:40.519845 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:40.519852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:40.519855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:40.520066 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:40.520226 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:40.520227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:40.520227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:40.520317 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:40.520577 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:40.520583 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:40.520770 (+ 187us) client_negotiation.cc:769] Sending connection context
1026 15:13:40.520926 (+ 156us) client_negotiation.cc:240] Negotiation successful
1026 15:13:40.521096 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20251026 15:13:40.521941 32654 heartbeater.cc:344] Connected to a master server at 127.14.212.126:43755
I20251026 15:13:40.522050 32654 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:40.522284 32654 heartbeater.cc:507] Master 127.14.212.126:43755 requested a full tablet report, sending...
I20251026 15:13:40.522804 32461 ts_manager.cc:194] Registered new tserver with Master: 5603764075d641dc9468cbd0a7b9f79d (127.14.212.65:45575)
I20251026 15:13:40.523432 32461 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:42709
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:40.564798 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:43755
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:40.645224 32661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:40.645430 32661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:40.645473 32661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:40.646993 32661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:40.647048 32661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:40.647095 32661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:13:40.648738 32661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:43755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:40.649021 32661 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:40.649224 32661 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:40.651912 32667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:40.652061 32666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:40.652100 32669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:40.652185 32661 server_base.cc:1047] running on GCE node
I20251026 15:13:40.652473 32661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:40.652655 32661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:40.653801 32661 hybrid_clock.cc:648] HybridClock initialized: now 1761491620653784 us; error 28 us; skew 500 ppm
Oct 26 15:13:40 dist-test-slave-8pgz krb5kdc[32413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491620, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:40.656461 32661 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:13:40.657016 32661 webserver.cc:492] Webserver started at http://127.14.212.66:44013/ using document root <none> and password file <none>
I20251026 15:13:40.657204 32661 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:40.657240 32661 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:40.657326 32661 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:40.658282 32661 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "60524c03a87c44bdb18aa43be7972bf0"
format_stamp: "Formatted at 2025-10-26 15:13:40 on dist-test-slave-8pgz"
server_key: "e00c07b54a1d5f857b6c96903b7e31b0"
server_key_iv: "4469d709bb8340661716cabde7d24f0b"
server_key_version: "encryptionkey@0"
I20251026 15:13:40.658638 32661 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "60524c03a87c44bdb18aa43be7972bf0"
format_stamp: "Formatted at 2025-10-26 15:13:40 on dist-test-slave-8pgz"
server_key: "e00c07b54a1d5f857b6c96903b7e31b0"
server_key_iv: "4469d709bb8340661716cabde7d24f0b"
server_key_version: "encryptionkey@0"
I20251026 15:13:40.659869 32661 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:40.660597 32676 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:40.660779 32661 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:13:40.660843 32661 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "60524c03a87c44bdb18aa43be7972bf0"
format_stamp: "Formatted at 2025-10-26 15:13:40 on dist-test-slave-8pgz"
server_key: "e00c07b54a1d5f857b6c96903b7e31b0"
server_key_iv: "4469d709bb8340661716cabde7d24f0b"
server_key_version: "encryptionkey@0"
I20251026 15:13:40.660892 32661 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:40.675143 32661 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:40.677142 32661 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:40.677256 32661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:40.677433 32661 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:40.677736 32661 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:40.677781 32661 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:40.677805 32661 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:40.677831 32661 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:40.683408 32661 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:35905
I20251026 15:13:40.683526 321 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:35905 every 8 connection(s)
I20251026 15:13:40.683758 32661 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
Oct 26 15:13:40 dist-test-slave-8pgz krb5kdc[32413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491620, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:40.689564 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 32661
I20251026 15:13:40.689666 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:13:40.689831 15185 external_mini_cluster.cc:1468] Setting key ca262d9f603775af5146bcba11541b9a
I20251026 15:13:40.692739 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.684463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:53773 (local address 127.14.212.126:43755)
1026 15:13:40.684565 (+ 102us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:40.684567 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:40.684735 (+ 168us) server_negotiation.cc:408] Connection header received
1026 15:13:40.685307 (+ 572us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:40.685308 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:40.685322 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:40.685381 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:40.686140 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.686669 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.687218 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.687394 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.689815 (+ 2421us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:40.689820 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:40.689820 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:40.689828 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:40.691545 (+ 1717us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:40.691906 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:40.691907 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:40.691907 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:40.691965 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:40.692152 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:40.692152 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:40.692153 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:40.692246 (+ 93us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:40.692306 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:40.692553 (+ 247us) server_negotiation.cc:300] Negotiation successful
1026 15:13:40.692652 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251026 15:13:40.692883 324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.684425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43755 (local address 127.14.212.66:53773)
1026 15:13:40.684617 (+ 192us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:40.684653 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:40.685209 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:40.685457 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:40.685461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:40.685769 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:40.686046 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.686049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.686759 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.686760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:40.687147 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.687149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.687274 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.688214 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:40.688229 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:40.689731 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:40.691628 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:40.691634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:40.691637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:40.691835 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:40.692024 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:40.692025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:40.692025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:40.692089 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:40.692300 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:40.692305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:40.692432 (+ 127us) client_negotiation.cc:769] Sending connection context
1026 15:13:40.692530 (+ 98us) client_negotiation.cc:240] Negotiation successful
1026 15:13:40.692672 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20251026 15:13:40.693320 322 heartbeater.cc:344] Connected to a master server at 127.14.212.126:43755
I20251026 15:13:40.693456 322 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:40.693720 322 heartbeater.cc:507] Master 127.14.212.126:43755 requested a full tablet report, sending...
I20251026 15:13:40.694170 32461 ts_manager.cc:194] Registered new tserver with Master: 60524c03a87c44bdb18aa43be7972bf0 (127.14.212.66:35905)
I20251026 15:13:40.694621 32461 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:53773
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:40.736975 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:43755
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:40.817572 329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:40.817900 329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:40.817966 329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:40.819497 329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:40.819548 329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:40.819576 329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:13:40.821226 329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:43755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:40.821465 329 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:40.821664 329 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:40.824322 337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:40.824368 334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:40.824465 329 server_base.cc:1047] running on GCE node
W20251026 15:13:40.824322 335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:40.824692 329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:40.824867 329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:40.825992 329 hybrid_clock.cc:648] HybridClock initialized: now 1761491620825982 us; error 33 us; skew 500 ppm
Oct 26 15:13:40 dist-test-slave-8pgz krb5kdc[32413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491620, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:40.828512 329 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:13:40.828997 329 webserver.cc:492] Webserver started at http://127.14.212.67:36979/ using document root <none> and password file <none>
I20251026 15:13:40.829173 329 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:40.829206 329 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:40.829295 329 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:40.830289 329 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "78d797243f7c4ca48f01ca110acd85e8"
format_stamp: "Formatted at 2025-10-26 15:13:40 on dist-test-slave-8pgz"
server_key: "b23614ea6f69a32f64a6cba676a8dcae"
server_key_iv: "5181cef6d379ce0bea3ab308d5a0e08d"
server_key_version: "encryptionkey@0"
I20251026 15:13:40.830642 329 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "78d797243f7c4ca48f01ca110acd85e8"
format_stamp: "Formatted at 2025-10-26 15:13:40 on dist-test-slave-8pgz"
server_key: "b23614ea6f69a32f64a6cba676a8dcae"
server_key_iv: "5181cef6d379ce0bea3ab308d5a0e08d"
server_key_version: "encryptionkey@0"
I20251026 15:13:40.831841 329 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:40.832630 344 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:40.832808 329 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:40.832873 329 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "78d797243f7c4ca48f01ca110acd85e8"
format_stamp: "Formatted at 2025-10-26 15:13:40 on dist-test-slave-8pgz"
server_key: "b23614ea6f69a32f64a6cba676a8dcae"
server_key_iv: "5181cef6d379ce0bea3ab308d5a0e08d"
server_key_version: "encryptionkey@0"
I20251026 15:13:40.832924 329 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:40.851303 329 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:40.853273 329 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:40.853385 329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:40.853561 329 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:40.853824 329 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:40.853853 329 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:40.853874 329 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:40.853888 329 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:40.859452 329 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:36845
I20251026 15:13:40.859541 457 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:36845 every 8 connection(s)
I20251026 15:13:40.859799 329 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:13:40.861543 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 329
I20251026 15:13:40.861620 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:13:40.861793 15185 external_mini_cluster.cc:1468] Setting key 981c3ec0454389054e8ce18c5c82f684
Oct 26 15:13:40 dist-test-slave-8pgz krb5kdc[32413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491620, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:40.868233 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.860487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:34305 (local address 127.14.212.126:43755)
1026 15:13:40.860563 (+ 76us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:40.860566 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:40.860666 (+ 100us) server_negotiation.cc:408] Connection header received
1026 15:13:40.861192 (+ 526us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:40.861193 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:40.861204 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:40.861262 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:40.862105 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.862790 (+ 685us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.863327 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.863447 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.865774 (+ 2327us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:40.865778 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:40.865778 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:40.865784 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:40.867152 (+ 1368us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:40.867511 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:40.867512 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:40.867513 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:40.867536 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:40.867701 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:40.867701 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:40.867701 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:40.867795 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:40.867847 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:40.868080 (+ 233us) server_negotiation.cc:300] Negotiation successful
1026 15:13:40.868162 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251026 15:13:40.868333 460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.860440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43755 (local address 127.14.212.67:34305)
1026 15:13:40.860563 (+ 123us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:40.860589 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:40.861085 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:40.861308 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:40.861311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:40.861643 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:40.862006 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.862011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.862855 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.862856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:40.863254 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.863255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.863359 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.864305 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:40.864320 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:40.865687 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:40.867226 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:40.867231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:40.867234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:40.867429 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:40.867595 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:40.867595 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:40.867596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:40.867644 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:40.867842 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:40.867845 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:40.867956 (+ 111us) client_negotiation.cc:769] Sending connection context
1026 15:13:40.868048 (+ 92us) client_negotiation.cc:240] Negotiation successful
1026 15:13:40.868152 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20251026 15:13:40.868834 458 heartbeater.cc:344] Connected to a master server at 127.14.212.126:43755
I20251026 15:13:40.868927 458 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:40.869144 458 heartbeater.cc:507] Master 127.14.212.126:43755 requested a full tablet report, sending...
I20251026 15:13:40.869527 32461 ts_manager.cc:194] Registered new tserver with Master: 78d797243f7c4ca48f01ca110acd85e8 (127.14.212.67:36845)
I20251026 15:13:40.869887 32461 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:34305
I20251026 15:13:40.873687 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:13:40.880332 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.875030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.14.212.126:43755)
1026 15:13:40.875075 (+ 45us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:40.875077 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:40.875179 (+ 102us) server_negotiation.cc:408] Connection header received
1026 15:13:40.875249 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:40.875250 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:40.875258 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:40.875298 (+ 40us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:40.875933 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.876405 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.876927 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.877039 (+ 112us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.878143 (+ 1104us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:40.878146 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:40.878146 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:40.878153 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:40.879546 (+ 1393us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:40.879830 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:40.879831 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:40.879831 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:40.879847 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:40.879943 (+ 96us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:40.879944 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:40.879944 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:40.880063 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:40.880136 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:40.880206 (+ 70us) server_negotiation.cc:300] Negotiation successful
1026 15:13:40.880268 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251026 15:13:40.881780 32461 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:13:40.883059 32461 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:13:40.887990 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.885184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:35905 (local address 127.0.0.1:46702)
1026 15:13:40.885301 (+ 117us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:40.885332 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:40.885409 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:40.885707 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:40.885710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:40.885712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:40.885988 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.885991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.886992 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.886993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:40.887670 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.887671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.887739 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.887776 (+ 37us) client_negotiation.cc:769] Sending connection context
1026 15:13:40.887827 (+ 51us) client_negotiation.cc:240] Negotiation successful
1026 15:13:40.887863 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
I20251026 15:13:40.888495 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.885196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:36845 (local address 127.0.0.1:49386)
1026 15:13:40.885742 (+ 546us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:40.885751 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:40.885806 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:40.885956 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:40.885957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:40.885958 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:40.886205 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.886207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.887493 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.887494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:40.888272 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.888275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.888348 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.888351 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:13:40.888384 (+ 33us) client_negotiation.cc:240] Negotiation successful
1026 15:13:40.888432 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":86,"threads_started":1}
I20251026 15:13:40.888713 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.885358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46702 (local address 127.14.212.66:35905)
1026 15:13:40.885527 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:40.885536 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:40.885548 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:13:40.885589 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:40.885591 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:40.885610 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:40.885688 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:40.886069 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.886884 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.887937 (+ 1053us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.888485 (+ 548us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.888546 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:40.888580 (+ 34us) server_negotiation.cc:300] Negotiation successful
1026 15:13:40.888632 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20251026 15:13:40.889032 471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.885189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:45575 (local address 127.0.0.1:48632)
1026 15:13:40.885596 (+ 407us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:40.885606 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:40.885661 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:40.885873 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:40.885874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:40.885875 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:40.886088 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.886090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.887449 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.887450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:40.888355 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.888358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.888424 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.888428 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:40.888463 (+ 35us) client_negotiation.cc:240] Negotiation successful
1026 15:13:40.888506 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":106,"threads_started":1}
I20251026 15:13:40.889343 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.885450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49386 (local address 127.14.212.67:36845)
1026 15:13:40.885601 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:40.885607 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:40.885820 (+ 213us) server_negotiation.cc:408] Connection header received
1026 15:13:40.885864 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:40.885866 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:40.885882 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:40.885943 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:40.886295 (+ 352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.887418 (+ 1123us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.888693 (+ 1275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.889170 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.889220 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:40.889250 (+ 30us) server_negotiation.cc:300] Negotiation successful
1026 15:13:40.889288 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20251026 15:13:40.889330 32724 tablet_service.cc:1505] Processing CreateTablet for tablet 077ad941486641ed9c5d748a008dcad3 (DEFAULT_TABLE table=test-table [id=ab6355f610f245b19227ac2b489e988e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:40.889650 32724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 077ad941486641ed9c5d748a008dcad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:40.889704 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.885425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48632 (local address 127.14.212.65:45575)
1026 15:13:40.885577 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:40.885585 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:40.885649 (+ 64us) server_negotiation.cc:408] Connection header received
1026 15:13:40.885725 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:40.885728 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:40.885745 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:40.885812 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:40.886182 (+ 370us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.887365 (+ 1183us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.888564 (+ 1199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.889504 (+ 940us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.889549 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:40.889580 (+ 31us) server_negotiation.cc:300] Negotiation successful
1026 15:13:40.889623 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20251026 15:13:40.890164 32588 tablet_service.cc:1505] Processing CreateTablet for tablet 077ad941486641ed9c5d748a008dcad3 (DEFAULT_TABLE table=test-table [id=ab6355f610f245b19227ac2b489e988e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:40.890451 32588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 077ad941486641ed9c5d748a008dcad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:40.890882 392 tablet_service.cc:1505] Processing CreateTablet for tablet 077ad941486641ed9c5d748a008dcad3 (DEFAULT_TABLE table=test-table [id=ab6355f610f245b19227ac2b489e988e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:40.891155 392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 077ad941486641ed9c5d748a008dcad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:40.892062 475 tablet_bootstrap.cc:492] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0: Bootstrap starting.
I20251026 15:13:40.892704 475 tablet_bootstrap.cc:654] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:40.892913 475 log.cc:826] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:40.893244 477 tablet_bootstrap.cc:492] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d: Bootstrap starting.
I20251026 15:13:40.893548 475 tablet_bootstrap.cc:492] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0: No bootstrap required, opened a new log
I20251026 15:13:40.893587 475 ts_tablet_manager.cc:1403] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:13:40.893849 478 tablet_bootstrap.cc:492] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8: Bootstrap starting.
I20251026 15:13:40.894454 478 tablet_bootstrap.cc:654] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:40.894827 478 log.cc:826] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:40.895001 477 tablet_bootstrap.cc:654] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:40.895275 477 log.cc:826] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:40.895391 475 raft_consensus.cc:359] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60524c03a87c44bdb18aa43be7972bf0" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35905 } } peers { permanent_uuid: "5603764075d641dc9468cbd0a7b9f79d" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 45575 } } peers { permanent_uuid: "78d797243f7c4ca48f01ca110acd85e8" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36845 } }
I20251026 15:13:40.895526 475 raft_consensus.cc:385] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:40.895572 475 raft_consensus.cc:740] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60524c03a87c44bdb18aa43be7972bf0, State: Initialized, Role: FOLLOWER
I20251026 15:13:40.895670 475 consensus_queue.cc:260] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60524c03a87c44bdb18aa43be7972bf0" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35905 } } peers { permanent_uuid: "5603764075d641dc9468cbd0a7b9f79d" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 45575 } } peers { permanent_uuid: "78d797243f7c4ca48f01ca110acd85e8" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36845 } }
I20251026 15:13:40.895948 322 heartbeater.cc:499] Master 127.14.212.126:43755 was elected leader, sending a full tablet report...
I20251026 15:13:40.895993 477 tablet_bootstrap.cc:492] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d: No bootstrap required, opened a new log
I20251026 15:13:40.896049 477 ts_tablet_manager.cc:1403] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251026 15:13:40.896081 475 ts_tablet_manager.cc:1434] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251026 15:13:40.896742 478 tablet_bootstrap.cc:492] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8: No bootstrap required, opened a new log
I20251026 15:13:40.896806 478 ts_tablet_manager.cc:1403] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251026 15:13:40.897593 477 raft_consensus.cc:359] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60524c03a87c44bdb18aa43be7972bf0" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35905 } } peers { permanent_uuid: "5603764075d641dc9468cbd0a7b9f79d" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 45575 } } peers { permanent_uuid: "78d797243f7c4ca48f01ca110acd85e8" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36845 } }
I20251026 15:13:40.897725 477 raft_consensus.cc:385] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:40.897774 477 raft_consensus.cc:740] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5603764075d641dc9468cbd0a7b9f79d, State: Initialized, Role: FOLLOWER
I20251026 15:13:40.897866 477 consensus_queue.cc:260] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60524c03a87c44bdb18aa43be7972bf0" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35905 } } peers { permanent_uuid: "5603764075d641dc9468cbd0a7b9f79d" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 45575 } } peers { permanent_uuid: "78d797243f7c4ca48f01ca110acd85e8" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36845 } }
I20251026 15:13:40.898161 32654 heartbeater.cc:499] Master 127.14.212.126:43755 was elected leader, sending a full tablet report...
I20251026 15:13:40.898294 477 ts_tablet_manager.cc:1434] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:13:40.898471 478 raft_consensus.cc:359] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60524c03a87c44bdb18aa43be7972bf0" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35905 } } peers { permanent_uuid: "5603764075d641dc9468cbd0a7b9f79d" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 45575 } } peers { permanent_uuid: "78d797243f7c4ca48f01ca110acd85e8" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36845 } }
I20251026 15:13:40.898599 478 raft_consensus.cc:385] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:40.898626 478 raft_consensus.cc:740] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d797243f7c4ca48f01ca110acd85e8, State: Initialized, Role: FOLLOWER
I20251026 15:13:40.898715 478 consensus_queue.cc:260] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60524c03a87c44bdb18aa43be7972bf0" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35905 } } peers { permanent_uuid: "5603764075d641dc9468cbd0a7b9f79d" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 45575 } } peers { permanent_uuid: "78d797243f7c4ca48f01ca110acd85e8" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36845 } }
I20251026 15:13:40.898952 478 ts_tablet_manager.cc:1434] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:13:40.898967 458 heartbeater.cc:499] Master 127.14.212.126:43755 was elected leader, sending a full tablet report...
W20251026 15:13:40.934298 323 tablet.cc:2378] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:13:40.975155 482 raft_consensus.cc:493] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:13:40.975317 482 raft_consensus.cc:515] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60524c03a87c44bdb18aa43be7972bf0" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35905 } } peers { permanent_uuid: "5603764075d641dc9468cbd0a7b9f79d" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 45575 } } peers { permanent_uuid: "78d797243f7c4ca48f01ca110acd85e8" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36845 } }
I20251026 15:13:40.975802 482 leader_election.cc:290] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60524c03a87c44bdb18aa43be7972bf0 (127.14.212.66:35905), 78d797243f7c4ca48f01ca110acd85e8 (127.14.212.67:36845)
I20251026 15:13:40.978363 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.976021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:36845 (local address 127.14.212.65:35283)
1026 15:13:40.976087 (+ 66us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:40.976097 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:40.976145 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:40.976361 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:40.976362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:40.976364 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:40.976597 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.976599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.977514 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.977516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:40.978113 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.978115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.978182 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.978186 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:40.978214 (+ 28us) client_negotiation.cc:240] Negotiation successful
1026 15:13:40.978261 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20251026 15:13:40.978468 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.976072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:35905 (local address 127.14.212.65:45559)
1026 15:13:40.976379 (+ 307us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:40.976389 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:40.976461 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:40.976621 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:40.976622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:40.976623 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:40.976852 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.976853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.977546 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.977547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:40.978147 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:40.978149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.978236 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.978240 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:40.978273 (+ 33us) client_negotiation.cc:240] Negotiation successful
1026 15:13:40.978310 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":249,"threads_started":1}
I20251026 15:13:40.978878 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.976162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:45559 (local address 127.14.212.66:35905)
1026 15:13:40.976304 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:40.976306 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:40.976442 (+ 136us) server_negotiation.cc:408] Connection header received
1026 15:13:40.976539 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:40.976542 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:40.976551 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:40.976605 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:40.976925 (+ 320us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.977422 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.978246 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.978778 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.978786 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:40.978807 (+ 21us) server_negotiation.cc:300] Negotiation successful
1026 15:13:40.978831 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20251026 15:13:40.978968 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:40.976131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:35283 (local address 127.14.212.67:36845)
1026 15:13:40.976188 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:40.976191 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:40.976210 (+ 19us) server_negotiation.cc:408] Connection header received
1026 15:13:40.976246 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:40.976247 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:40.976259 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:40.976657 (+ 398us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:40.976694 (+ 37us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.977424 (+ 730us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:40.978240 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:40.978842 (+ 602us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:40.978851 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:40.978877 (+ 26us) server_negotiation.cc:300] Negotiation successful
1026 15:13:40.978904 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251026 15:13:40.979207 412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "077ad941486641ed9c5d748a008dcad3" candidate_uuid: "5603764075d641dc9468cbd0a7b9f79d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d797243f7c4ca48f01ca110acd85e8" is_pre_election: true
I20251026 15:13:40.979204 32744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "077ad941486641ed9c5d748a008dcad3" candidate_uuid: "5603764075d641dc9468cbd0a7b9f79d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60524c03a87c44bdb18aa43be7972bf0" is_pre_election: true
I20251026 15:13:40.979368 412 raft_consensus.cc:2468] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5603764075d641dc9468cbd0a7b9f79d in term 0.
I20251026 15:13:40.979369 32744 raft_consensus.cc:2468] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5603764075d641dc9468cbd0a7b9f79d in term 0.
I20251026 15:13:40.979583 32541 leader_election.cc:304] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5603764075d641dc9468cbd0a7b9f79d, 78d797243f7c4ca48f01ca110acd85e8; no voters:
I20251026 15:13:40.979748 482 raft_consensus.cc:2804] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:13:40.979807 482 raft_consensus.cc:493] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:13:40.979842 482 raft_consensus.cc:3060] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:40.980507 482 raft_consensus.cc:515] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60524c03a87c44bdb18aa43be7972bf0" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35905 } } peers { permanent_uuid: "5603764075d641dc9468cbd0a7b9f79d" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 45575 } } peers { permanent_uuid: "78d797243f7c4ca48f01ca110acd85e8" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36845 } }
I20251026 15:13:40.980643 482 leader_election.cc:290] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [CANDIDATE]: Term 1 election: Requested vote from peers 60524c03a87c44bdb18aa43be7972bf0 (127.14.212.66:35905), 78d797243f7c4ca48f01ca110acd85e8 (127.14.212.67:36845)
I20251026 15:13:40.980794 412 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "077ad941486641ed9c5d748a008dcad3" candidate_uuid: "5603764075d641dc9468cbd0a7b9f79d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d797243f7c4ca48f01ca110acd85e8"
I20251026 15:13:40.980831 32744 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "077ad941486641ed9c5d748a008dcad3" candidate_uuid: "5603764075d641dc9468cbd0a7b9f79d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60524c03a87c44bdb18aa43be7972bf0"
I20251026 15:13:40.980871 412 raft_consensus.cc:3060] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:40.980913 32744 raft_consensus.cc:3060] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:40.981564 412 raft_consensus.cc:2468] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5603764075d641dc9468cbd0a7b9f79d in term 1.
I20251026 15:13:40.981638 32744 raft_consensus.cc:2468] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5603764075d641dc9468cbd0a7b9f79d in term 1.
I20251026 15:13:40.981738 32541 leader_election.cc:304] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5603764075d641dc9468cbd0a7b9f79d, 78d797243f7c4ca48f01ca110acd85e8; no voters:
I20251026 15:13:40.981858 482 raft_consensus.cc:2804] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:40.982045 482 raft_consensus.cc:697] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [term 1 LEADER]: Becoming Leader. State: Replica: 5603764075d641dc9468cbd0a7b9f79d, State: Running, Role: LEADER
I20251026 15:13:40.982156 482 consensus_queue.cc:237] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60524c03a87c44bdb18aa43be7972bf0" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35905 } } peers { permanent_uuid: "5603764075d641dc9468cbd0a7b9f79d" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 45575 } } peers { permanent_uuid: "78d797243f7c4ca48f01ca110acd85e8" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36845 } }
I20251026 15:13:40.982973 32460 catalog_manager.cc:5649] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5603764075d641dc9468cbd0a7b9f79d (127.14.212.65). New cstate: current_term: 1 leader_uuid: "5603764075d641dc9468cbd0a7b9f79d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60524c03a87c44bdb18aa43be7972bf0" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5603764075d641dc9468cbd0a7b9f79d" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 45575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78d797243f7c4ca48f01ca110acd85e8" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36845 } health_report { overall_health: UNKNOWN } } }
I20251026 15:13:41.006788 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.004857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.14.212.65:45575)
1026 15:13:41.004911 (+ 54us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:41.004913 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:41.004952 (+ 39us) server_negotiation.cc:408] Connection header received
1026 15:13:41.004998 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:41.004999 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:41.005015 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:41.005077 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:13:41.005295 (+ 218us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.005752 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.006276 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.006446 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.006493 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:13:41.006586 (+ 93us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:13:41.006652 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:41.006704 (+ 52us) server_negotiation.cc:300] Negotiation successful
1026 15:13:41.006728 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251026 15:13:41.008442 412 raft_consensus.cc:1275] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8 [term 1 FOLLOWER]: Refusing update from remote peer 5603764075d641dc9468cbd0a7b9f79d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:41.008442 32744 raft_consensus.cc:1275] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0 [term 1 FOLLOWER]: Refusing update from remote peer 5603764075d641dc9468cbd0a7b9f79d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:41.008728 482 consensus_queue.cc:1048] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78d797243f7c4ca48f01ca110acd85e8" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 36845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:41.008845 485 consensus_queue.cc:1048] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [LEADER]: Connected to new peer: Peer: permanent_uuid: "60524c03a87c44bdb18aa43be7972bf0" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:41.012048 490 mvcc.cc:204] Tried to move back new op lower bound from 7215069679648059392 to 7215069679543513088. Current Snapshot: MvccSnapshot[applied={T|T < 7215069679648059392}]
I20251026 15:13:41.012179 489 mvcc.cc:204] Tried to move back new op lower bound from 7215069679648059392 to 7215069679543513088. Current Snapshot: MvccSnapshot[applied={T|T < 7215069679648059392 or (T in {7215069679648059392})}]
I20251026 15:13:41.013370 32461 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48536:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:41.013468 32461 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48536:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:41.014231 32461 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 70fb1e153c5f4011ad49c5071b02acda: Sending DeleteTablet for 3 replicas of tablet 077ad941486641ed9c5d748a008dcad3
I20251026 15:13:41.014530 32588 tablet_service.cc:1552] Processing DeleteTablet for tablet 077ad941486641ed9c5d748a008dcad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:48632
I20251026 15:13:41.014601 392 tablet_service.cc:1552] Processing DeleteTablet for tablet 077ad941486641ed9c5d748a008dcad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:49386
I20251026 15:13:41.014597 32724 tablet_service.cc:1552] Processing DeleteTablet for tablet 077ad941486641ed9c5d748a008dcad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:41 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:46702
I20251026 15:13:41.014760 498 tablet_replica.cc:333] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d: stopping tablet replica
I20251026 15:13:41.014833 498 raft_consensus.cc:2243] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:13:41.014844 500 tablet_replica.cc:333] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8: stopping tablet replica
I20251026 15:13:41.014971 498 raft_consensus.cc:2272] T 077ad941486641ed9c5d748a008dcad3 P 5603764075d641dc9468cbd0a7b9f79d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:41.015050 500 raft_consensus.cc:2243] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:41.015165 500 raft_consensus.cc:2272] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:41.015220 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 32525
I20251026 15:13:41.015822 499 tablet_replica.cc:333] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0: stopping tablet replica
I20251026 15:13:41.015904 499 raft_consensus.cc:2243] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:41.016012 499 raft_consensus.cc:2272] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:41.016129 500 ts_tablet_manager.cc:1916] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:41.016479 499 ts_tablet_manager.cc:1916] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:41.017652 500 ts_tablet_manager.cc:1929] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:41.017702 500 log.cc:1199] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/077ad941486641ed9c5d748a008dcad3
I20251026 15:13:41.017930 500 ts_tablet_manager.cc:1950] T 077ad941486641ed9c5d748a008dcad3 P 78d797243f7c4ca48f01ca110acd85e8: Deleting consensus metadata
I20251026 15:13:41.018306 32445 catalog_manager.cc:4985] TS 78d797243f7c4ca48f01ca110acd85e8 (127.14.212.67:36845): tablet 077ad941486641ed9c5d748a008dcad3 (table test-table [id=ab6355f610f245b19227ac2b489e988e]) successfully deleted
I20251026 15:13:41.018656 499 ts_tablet_manager.cc:1929] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:41.018720 499 log.cc:1199] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/077ad941486641ed9c5d748a008dcad3
I20251026 15:13:41.019011 499 ts_tablet_manager.cc:1950] T 077ad941486641ed9c5d748a008dcad3 P 60524c03a87c44bdb18aa43be7972bf0: Deleting consensus metadata
I20251026 15:13:41.019395 32446 catalog_manager.cc:4985] TS 60524c03a87c44bdb18aa43be7972bf0 (127.14.212.66:35905): tablet 077ad941486641ed9c5d748a008dcad3 (table test-table [id=ab6355f610f245b19227ac2b489e988e]) successfully deleted
W20251026 15:13:41.023339 32447 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:45575 (error 108)
I20251026 15:13:41.023550 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 32661
I20251026 15:13:41.024338 471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.024213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:45575 (local address 127.0.0.1:48648)
1026 15:13:41.024266 (+ 53us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:41.024303 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:45575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20251026 15:13:41.024444 32447 catalog_manager.cc:4712] TS 5603764075d641dc9468cbd0a7b9f79d (127.14.212.65:45575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 077ad941486641ed9c5d748a008dcad3: Network error: Client connection negotiation failed: client connection to 127.14.212.65:45575: connect: Connection refused (error 111)
I20251026 15:13:41.028538 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 329
I20251026 15:13:41.033457 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 32429
2025-10-26T15:13:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (3022 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2025-10-26T15:13:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:13:41Z Disabled control of system clock
I20251026 15:13:41.060911 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:45387
--webserver_interface=127.14.212.126
--webserver_port=0
--builtin_ntp_servers=127.14.212.84:36195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:45387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251026 15:13:41.140133 505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:41.140327 505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:41.140349 505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:41.141856 505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:13:41.141914 505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:41.141928 505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:41.141940 505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:13:41.141952 505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:13:41.143572 505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:45387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:45387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:41.143833 505 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:41.144088 505 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:41.146692 511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:41.146708 510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:41.146828 513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:41.146955 505 server_base.cc:1047] running on GCE node
I20251026 15:13:41.147166 505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:41.147413 505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:41.148591 505 hybrid_clock.cc:648] HybridClock initialized: now 1761491621148559 us; error 50 us; skew 500 ppm
I20251026 15:13:41.149758 505 webserver.cc:492] Webserver started at http://127.14.212.126:43077/ using document root <none> and password file <none>
I20251026 15:13:41.149973 505 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:41.150024 505 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:41.150144 505 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:41.151176 505 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "52e5430d27b74e63b686a9ba8a50cdf4"
format_stamp: "Formatted at 2025-10-26 15:13:41 on dist-test-slave-8pgz"
server_key: "9ef8a888f25aa42ad9f1ba918f520d24"
server_key_iv: "48c2b386bf54648f2c4f061411224d56"
server_key_version: "encryptionkey@0"
I20251026 15:13:41.151487 505 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "52e5430d27b74e63b686a9ba8a50cdf4"
format_stamp: "Formatted at 2025-10-26 15:13:41 on dist-test-slave-8pgz"
server_key: "9ef8a888f25aa42ad9f1ba918f520d24"
server_key_iv: "48c2b386bf54648f2c4f061411224d56"
server_key_version: "encryptionkey@0"
I20251026 15:13:41.152701 505 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:13:41.153487 519 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:41.153726 505 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:41.153810 505 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "52e5430d27b74e63b686a9ba8a50cdf4"
format_stamp: "Formatted at 2025-10-26 15:13:41 on dist-test-slave-8pgz"
server_key: "9ef8a888f25aa42ad9f1ba918f520d24"
server_key_iv: "48c2b386bf54648f2c4f061411224d56"
server_key_version: "encryptionkey@0"
I20251026 15:13:41.153875 505 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:41.187518 505 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:41.187826 505 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:41.187958 505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:41.191893 505 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:45387
I20251026 15:13:41.191939 571 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:45387 every 8 connection(s)
I20251026 15:13:41.192269 505 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:13:41.192847 572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:41.195302 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 505
I20251026 15:13:41.195403 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:13:41.195564 15185 external_mini_cluster.cc:1468] Setting key b4d282a2d8708e00f3db90bba578270e
I20251026 15:13:41.195755 572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4: Bootstrap starting.
I20251026 15:13:41.196506 572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:41.196805 572 log.cc:826] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:41.197614 572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4: No bootstrap required, opened a new log
I20251026 15:13:41.199043 572 raft_consensus.cc:359] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e5430d27b74e63b686a9ba8a50cdf4" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45387 } }
I20251026 15:13:41.199170 572 raft_consensus.cc:385] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:41.199204 572 raft_consensus.cc:740] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e5430d27b74e63b686a9ba8a50cdf4, State: Initialized, Role: FOLLOWER
I20251026 15:13:41.199353 572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e5430d27b74e63b686a9ba8a50cdf4" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45387 } }
I20251026 15:13:41.199741 572 raft_consensus.cc:399] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:13:41.199797 572 raft_consensus.cc:493] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:13:41.199848 572 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:41.200474 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.196064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41048 (local address 127.14.212.126:45387)
1026 15:13:41.196305 (+ 241us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:41.196312 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:41.196334 (+ 22us) server_negotiation.cc:408] Connection header received
1026 15:13:41.196774 (+ 440us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:41.196783 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:41.196854 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:41.196994 (+ 140us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:41.197449 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.198604 (+ 1155us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.199232 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.199517 (+ 285us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.199886 (+ 369us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:41.199895 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:41.199895 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:41.200102 (+ 207us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:41.200128 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:41.200136 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:41.200220 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:41.200294 (+ 74us) server_negotiation.cc:300] Negotiation successful
1026 15:13:41.200349 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20251026 15:13:41.200466 572 raft_consensus.cc:515] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e5430d27b74e63b686a9ba8a50cdf4" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45387 } }
I20251026 15:13:41.200640 572 leader_election.cc:304] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52e5430d27b74e63b686a9ba8a50cdf4; no voters:
I20251026 15:13:41.200807 572 leader_election.cc:290] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:13:41.200892 577 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:41.201087 577 raft_consensus.cc:697] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [term 1 LEADER]: Becoming Leader. State: Replica: 52e5430d27b74e63b686a9ba8a50cdf4, State: Running, Role: LEADER
I20251026 15:13:41.201086 572 sys_catalog.cc:565] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:13:41.201299 577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e5430d27b74e63b686a9ba8a50cdf4" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45387 } }
I20251026 15:13:41.201673 578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52e5430d27b74e63b686a9ba8a50cdf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e5430d27b74e63b686a9ba8a50cdf4" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45387 } } }
I20251026 15:13:41.201711 579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52e5430d27b74e63b686a9ba8a50cdf4. Latest consensus state: current_term: 1 leader_uuid: "52e5430d27b74e63b686a9ba8a50cdf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e5430d27b74e63b686a9ba8a50cdf4" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45387 } } }
I20251026 15:13:41.201828 578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:41.201859 579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:41.202245 585 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:13:41.202718 585 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:13:41.204187 585 catalog_manager.cc:1357] Generated new cluster ID: a4b0217aa8cc48c6b5400232166c0fe2
I20251026 15:13:41.204236 585 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:13:41.214311 585 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:13:41.215145 585 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:13:41.222440 585 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4: Generated new TSK 0
I20251026 15:13:41.222662 585 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:13:41.225600 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:45387
--builtin_ntp_servers=127.14.212.84:36195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251026 15:13:41.307708 596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:41.307880 596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:41.307922 596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:41.309296 596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:41.309341 596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:41.309379 596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:13:41.310899 596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:45387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:41.311170 596 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:41.311396 596 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:41.313903 601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:41.313890 602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:41.313890 604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:41.314214 596 server_base.cc:1047] running on GCE node
I20251026 15:13:41.314370 596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:41.314601 596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:41.315766 596 hybrid_clock.cc:648] HybridClock initialized: now 1761491621315750 us; error 36 us; skew 500 ppm
I20251026 15:13:41.316994 596 webserver.cc:492] Webserver started at http://127.14.212.65:44637/ using document root <none> and password file <none>
I20251026 15:13:41.317201 596 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:41.317251 596 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:41.317378 596 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:41.318360 596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "f5822b17117b4b2fb98dead60b122495"
format_stamp: "Formatted at 2025-10-26 15:13:41 on dist-test-slave-8pgz"
server_key: "2431cac5988b973907bfb8c06717ad1c"
server_key_iv: "0598569f20e344aa2ecab415887e69c2"
server_key_version: "encryptionkey@0"
I20251026 15:13:41.318745 596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "f5822b17117b4b2fb98dead60b122495"
format_stamp: "Formatted at 2025-10-26 15:13:41 on dist-test-slave-8pgz"
server_key: "2431cac5988b973907bfb8c06717ad1c"
server_key_iv: "0598569f20e344aa2ecab415887e69c2"
server_key_version: "encryptionkey@0"
I20251026 15:13:41.319955 596 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:41.320628 610 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:41.320824 596 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:13:41.320917 596 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "f5822b17117b4b2fb98dead60b122495"
format_stamp: "Formatted at 2025-10-26 15:13:41 on dist-test-slave-8pgz"
server_key: "2431cac5988b973907bfb8c06717ad1c"
server_key_iv: "0598569f20e344aa2ecab415887e69c2"
server_key_version: "encryptionkey@0"
I20251026 15:13:41.320999 596 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:41.333285 596 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:41.333627 596 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:41.333775 596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:41.334026 596 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:41.334416 596 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:41.334465 596 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:41.334498 596 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:41.334529 596 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:41.341464 596 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:41315
I20251026 15:13:41.341518 723 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:41315 every 8 connection(s)
I20251026 15:13:41.341835 596 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:13:41.346203 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.342501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:48705 (local address 127.14.212.126:45387)
1026 15:13:41.342561 (+ 60us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:41.342564 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:41.342795 (+ 231us) server_negotiation.cc:408] Connection header received
1026 15:13:41.343279 (+ 484us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:41.343280 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:41.343292 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:41.343371 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:41.343804 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.344577 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.345126 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.345316 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.345924 (+ 608us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:41.345928 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:41.345929 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:41.345934 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:41.345950 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:41.345954 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:41.346012 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:41.346085 (+ 73us) server_negotiation.cc:300] Negotiation successful
1026 15:13:41.346119 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251026 15:13:41.346261 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.342506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:45387 (local address 127.14.212.65:48705)
1026 15:13:41.342694 (+ 188us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:41.342726 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:41.343193 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:41.343424 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:41.343427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:41.343435 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:41.343735 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.343737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.344662 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.344662 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:41.345056 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.345058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.345152 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.345811 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:13:41.345811 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:41.345831 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:13:41.345849 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:41.346016 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:41.346019 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:41.346021 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:13:41.346073 (+ 52us) client_negotiation.cc:240] Negotiation successful
1026 15:13:41.346135 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20251026 15:13:41.346905 724 heartbeater.cc:344] Connected to a master server at 127.14.212.126:45387
I20251026 15:13:41.347009 724 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:41.347193 724 heartbeater.cc:507] Master 127.14.212.126:45387 requested a full tablet report, sending...
I20251026 15:13:41.347601 534 ts_manager.cc:194] Registered new tserver with Master: f5822b17117b4b2fb98dead60b122495 (127.14.212.65:41315)
I20251026 15:13:41.348229 534 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.14.212.65:48705
I20251026 15:13:41.350313 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 596
I20251026 15:13:41.350463 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:13:41.350723 15185 external_mini_cluster.cc:1468] Setting key 0e1be0efb2a1bd132d9592ea4d3d8736
I20251026 15:13:41.351684 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:45387
--builtin_ntp_servers=127.14.212.84:36195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251026 15:13:41.434017 727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:41.434197 727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:41.434219 727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:41.435618 727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:41.435658 727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:41.435683 727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:13:41.437182 727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:45387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:41.437403 727 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:41.437594 727 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:41.440418 732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:41.440434 733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:41.440510 727 server_base.cc:1047] running on GCE node
W20251026 15:13:41.440608 735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:41.440791 727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:41.441004 727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:41.442145 727 hybrid_clock.cc:648] HybridClock initialized: now 1761491621442142 us; error 34 us; skew 500 ppm
I20251026 15:13:41.443362 727 webserver.cc:492] Webserver started at http://127.14.212.66:37395/ using document root <none> and password file <none>
I20251026 15:13:41.443562 727 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:41.443607 727 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:41.443732 727 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:41.444684 727 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "4143869438d8444f9c6ade219e1a6d92"
format_stamp: "Formatted at 2025-10-26 15:13:41 on dist-test-slave-8pgz"
server_key: "40780f8508ac4b1850dd5e7ed7b05447"
server_key_iv: "ffabb95ff7292c05f469fa164a733d0c"
server_key_version: "encryptionkey@0"
I20251026 15:13:41.444986 727 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "4143869438d8444f9c6ade219e1a6d92"
format_stamp: "Formatted at 2025-10-26 15:13:41 on dist-test-slave-8pgz"
server_key: "40780f8508ac4b1850dd5e7ed7b05447"
server_key_iv: "ffabb95ff7292c05f469fa164a733d0c"
server_key_version: "encryptionkey@0"
I20251026 15:13:41.446154 727 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:13:41.447022 741 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:41.447249 727 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:41.447322 727 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "4143869438d8444f9c6ade219e1a6d92"
format_stamp: "Formatted at 2025-10-26 15:13:41 on dist-test-slave-8pgz"
server_key: "40780f8508ac4b1850dd5e7ed7b05447"
server_key_iv: "ffabb95ff7292c05f469fa164a733d0c"
server_key_version: "encryptionkey@0"
I20251026 15:13:41.447386 727 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:41.464852 727 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:41.465178 727 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:41.465309 727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:41.465528 727 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:41.465823 727 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:41.465857 727 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:41.465893 727 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:41.465919 727 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:41.471540 727 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:34359
I20251026 15:13:41.471602 854 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:34359 every 8 connection(s)
I20251026 15:13:41.471896 727 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:13:41.476357 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.472621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:35687 (local address 127.14.212.126:45387)
1026 15:13:41.472684 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:41.472687 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:41.472877 (+ 190us) server_negotiation.cc:408] Connection header received
1026 15:13:41.473399 (+ 522us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:41.473401 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:41.473414 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:41.473471 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:41.473978 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.474524 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.475095 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.475296 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.475906 (+ 610us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:41.475910 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:41.475910 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:41.475915 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:41.476022 (+ 107us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:41.476034 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:41.476117 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:41.476237 (+ 120us) server_negotiation.cc:300] Negotiation successful
1026 15:13:41.476277 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251026 15:13:41.476526 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.472586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:45387 (local address 127.14.212.66:35687)
1026 15:13:41.472772 (+ 186us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:41.472797 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:41.473301 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:41.473536 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:41.473539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:41.473548 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:41.473855 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.473858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.474600 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.474602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:41.475010 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.475012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.475154 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.475782 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:13:41.475783 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:41.475805 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:13:41.475823 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:41.476135 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:41.476142 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:41.476145 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:13:41.476233 (+ 88us) client_negotiation.cc:240] Negotiation successful
1026 15:13:41.476343 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20251026 15:13:41.476550 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 727
I20251026 15:13:41.476662 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:13:41.476876 15185 external_mini_cluster.cc:1468] Setting key 6a5225af228661327af77454fd9a7e6d
I20251026 15:13:41.476998 855 heartbeater.cc:344] Connected to a master server at 127.14.212.126:45387
I20251026 15:13:41.477105 855 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:41.477326 855 heartbeater.cc:507] Master 127.14.212.126:45387 requested a full tablet report, sending...
I20251026 15:13:41.477656 534 ts_manager.cc:194] Registered new tserver with Master: 4143869438d8444f9c6ade219e1a6d92 (127.14.212.66:34359)
I20251026 15:13:41.477972 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:45387
--builtin_ntp_servers=127.14.212.84:36195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251026 15:13:41.478191 534 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.14.212.66:35687
W20251026 15:13:41.557902 858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:41.558081 858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:41.558105 858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:41.559604 858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:41.559655 858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:41.559681 858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:13:41.561136 858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:45387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:41.561349 858 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:41.561555 858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:41.564321 864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:41.564528 858 server_base.cc:1047] running on GCE node
W20251026 15:13:41.564340 863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:41.564333 866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:41.564808 858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:41.565006 858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:41.566143 858 hybrid_clock.cc:648] HybridClock initialized: now 1761491621566130 us; error 27 us; skew 500 ppm
I20251026 15:13:41.567258 858 webserver.cc:492] Webserver started at http://127.14.212.67:33365/ using document root <none> and password file <none>
I20251026 15:13:41.567458 858 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:41.567504 858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:41.567734 858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:41.568703 858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "d383a6e623c1447e9aafeb1530ca5fb0"
format_stamp: "Formatted at 2025-10-26 15:13:41 on dist-test-slave-8pgz"
server_key: "e93c0020586141bdc418b973a579713a"
server_key_iv: "1f0b01aac7117ff5225c2907a0f8d07d"
server_key_version: "encryptionkey@0"
I20251026 15:13:41.569029 858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "d383a6e623c1447e9aafeb1530ca5fb0"
format_stamp: "Formatted at 2025-10-26 15:13:41 on dist-test-slave-8pgz"
server_key: "e93c0020586141bdc418b973a579713a"
server_key_iv: "1f0b01aac7117ff5225c2907a0f8d07d"
server_key_version: "encryptionkey@0"
I20251026 15:13:41.570190 858 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:13:41.570979 872 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:41.571171 858 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251026 15:13:41.571246 858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "d383a6e623c1447e9aafeb1530ca5fb0"
format_stamp: "Formatted at 2025-10-26 15:13:41 on dist-test-slave-8pgz"
server_key: "e93c0020586141bdc418b973a579713a"
server_key_iv: "1f0b01aac7117ff5225c2907a0f8d07d"
server_key_version: "encryptionkey@0"
I20251026 15:13:41.571312 858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:41.591574 858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:41.591899 858 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:41.592039 858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:41.592259 858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:41.592582 858 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:41.592615 858 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:41.592638 858 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:41.592650 858 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:41.598345 858 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:46437
I20251026 15:13:41.598485 985 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:46437 every 8 connection(s)
I20251026 15:13:41.598734 858 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:13:41.602315 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 858
I20251026 15:13:41.602435 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:13:41.602595 15185 external_mini_cluster.cc:1468] Setting key c3162a0a724b6b97ee3293598f535b10
I20251026 15:13:41.603319 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.599384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:45387 (local address 127.14.212.67:50711)
1026 15:13:41.599564 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:41.599596 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:41.600064 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:41.600308 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:41.600311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:41.600319 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:41.600623 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.600625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.601262 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.601263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:41.601639 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.601640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.601772 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.602772 (+ 1000us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:13:41.602773 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:41.602792 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:13:41.602808 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:41.603018 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:41.603023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:41.603027 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:41.603107 (+ 80us) client_negotiation.cc:240] Negotiation successful
1026 15:13:41.603172 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20251026 15:13:41.603349 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.599429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:50711 (local address 127.14.212.126:45387)
1026 15:13:41.599487 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:41.599488 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:41.599662 (+ 174us) server_negotiation.cc:408] Connection header received
1026 15:13:41.600166 (+ 504us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:41.600168 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:41.600178 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:41.600242 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:41.600682 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.601192 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.601713 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.601892 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.602934 (+ 1042us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:41.602938 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:41.602938 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:41.602943 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:41.602959 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:41.602965 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:41.603018 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:41.603237 (+ 219us) server_negotiation.cc:300] Negotiation successful
1026 15:13:41.603270 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251026 15:13:41.604058 986 heartbeater.cc:344] Connected to a master server at 127.14.212.126:45387
I20251026 15:13:41.604154 986 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:41.604329 986 heartbeater.cc:507] Master 127.14.212.126:45387 requested a full tablet report, sending...
I20251026 15:13:41.604682 534 ts_manager.cc:194] Registered new tserver with Master: d383a6e623c1447e9aafeb1530ca5fb0 (127.14.212.67:46437)
I20251026 15:13:41.605043 534 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.14.212.67:50711
I20251026 15:13:41.614959 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:13:41.618803 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.616407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.14.212.126:45387)
1026 15:13:41.616469 (+ 62us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:41.616471 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:41.616480 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:13:41.616529 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:41.616529 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:41.616538 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:41.616594 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:41.616880 (+ 286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.617393 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.617933 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.618036 (+ 103us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.618544 (+ 508us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:41.618547 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:41.618547 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:41.618551 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:41.618562 (+ 11us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:41.618565 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:41.618610 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:41.618716 (+ 106us) server_negotiation.cc:300] Negotiation successful
1026 15:13:41.618744 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251026 15:13:41.620293 534 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:13:41.621584 534 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:13:41.626784 997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.623640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:46437 (local address 127.0.0.1:56102)
1026 15:13:41.623971 (+ 331us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:41.623982 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:41.624030 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:41.624339 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:41.624340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:41.624341 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:41.624535 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.624536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.625795 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.625796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:41.626485 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.626487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.626557 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.626594 (+ 37us) client_negotiation.cc:769] Sending connection context
1026 15:13:41.626646 (+ 52us) client_negotiation.cc:240] Negotiation successful
1026 15:13:41.626699 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":69,"threads_started":1}
I20251026 15:13:41.627424 994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.623637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:41315 (local address 127.0.0.1:41322)
1026 15:13:41.623761 (+ 124us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:41.623792 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:41.623865 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:41.624262 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:41.624265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:41.624266 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:41.624511 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.624513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.626270 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.626271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:41.627233 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.627236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.627298 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.627301 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:13:41.627331 (+ 30us) client_negotiation.cc:240] Negotiation successful
1026 15:13:41.627362 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20251026 15:13:41.627571 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.623989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.14.212.67:46437)
1026 15:13:41.624170 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:41.624182 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:41.624191 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:13:41.624226 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:41.624228 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:41.624253 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:41.624315 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:41.624613 (+ 298us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.625716 (+ 1103us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.626912 (+ 1196us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.627343 (+ 431us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.627421 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:41.627445 (+ 24us) server_negotiation.cc:300] Negotiation successful
1026 15:13:41.627469 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
I20251026 15:13:41.627637 995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.623842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:34359 (local address 127.0.0.1:58482)
1026 15:13:41.624407 (+ 565us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:41.624416 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:41.624469 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:41.624702 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:41.624703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:41.624705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:41.624960 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.624962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.626626 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.626627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:41.627437 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.627439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.627506 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.627510 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:41.627546 (+ 36us) client_negotiation.cc:240] Negotiation successful
1026 15:13:41.627586 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":69,"threads_started":1}
I20251026 15:13:41.628238 996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.623742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41322 (local address 127.14.212.65:41315)
1026 15:13:41.624063 (+ 321us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:41.624071 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:41.624084 (+ 13us) server_negotiation.cc:408] Connection header received
1026 15:13:41.624128 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:41.624130 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:41.624149 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:41.624239 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:41.624605 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.626174 (+ 1569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.627611 (+ 1437us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.628073 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.628136 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:41.628161 (+ 25us) server_negotiation.cc:300] Negotiation successful
1026 15:13:41.628184 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":97,"threads_started":1}
I20251026 15:13:41.628211 920 tablet_service.cc:1505] Processing CreateTablet for tablet c01a7fc0e62f42978e4bbe9b91a11ebc (DEFAULT_TABLE table=test-table [id=4e8b0729156d47a58865a46082a73101]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:41.628535 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.624021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58482 (local address 127.14.212.66:34359)
1026 15:13:41.624187 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:41.624195 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:41.624483 (+ 288us) server_negotiation.cc:408] Connection header received
1026 15:13:41.624531 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:41.624533 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:41.624549 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:41.624612 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:41.625054 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.626541 (+ 1487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.627834 (+ 1293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.628315 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.628386 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:41.628416 (+ 30us) server_negotiation.cc:300] Negotiation successful
1026 15:13:41.628455 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20251026 15:13:41.628546 920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c01a7fc0e62f42978e4bbe9b91a11ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:41.628716 658 tablet_service.cc:1505] Processing CreateTablet for tablet c01a7fc0e62f42978e4bbe9b91a11ebc (DEFAULT_TABLE table=test-table [id=4e8b0729156d47a58865a46082a73101]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:41.628976 658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c01a7fc0e62f42978e4bbe9b91a11ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:41.630769 789 tablet_service.cc:1505] Processing CreateTablet for tablet c01a7fc0e62f42978e4bbe9b91a11ebc (DEFAULT_TABLE table=test-table [id=4e8b0729156d47a58865a46082a73101]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:41.630924 1000 tablet_bootstrap.cc:492] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0: Bootstrap starting.
I20251026 15:13:41.631037 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c01a7fc0e62f42978e4bbe9b91a11ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:41.631238 1001 tablet_bootstrap.cc:492] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495: Bootstrap starting.
I20251026 15:13:41.631593 1000 tablet_bootstrap.cc:654] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:41.631803 1001 tablet_bootstrap.cc:654] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:41.631899 1000 log.cc:826] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:41.632076 1001 log.cc:826] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:41.632645 1000 tablet_bootstrap.cc:492] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0: No bootstrap required, opened a new log
I20251026 15:13:41.632688 1000 ts_tablet_manager.cc:1403] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:13:41.632752 1001 tablet_bootstrap.cc:492] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495: No bootstrap required, opened a new log
I20251026 15:13:41.632807 1001 ts_tablet_manager.cc:1403] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:13:41.633774 1004 tablet_bootstrap.cc:492] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92: Bootstrap starting.
I20251026 15:13:41.633873 1000 raft_consensus.cc:359] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46437 } } peers { permanent_uuid: "f5822b17117b4b2fb98dead60b122495" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41315 } } peers { permanent_uuid: "4143869438d8444f9c6ade219e1a6d92" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34359 } }
I20251026 15:13:41.634034 1000 raft_consensus.cc:385] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:41.634078 1000 raft_consensus.cc:740] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d383a6e623c1447e9aafeb1530ca5fb0, State: Initialized, Role: FOLLOWER
I20251026 15:13:41.634176 1000 consensus_queue.cc:260] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46437 } } peers { permanent_uuid: "f5822b17117b4b2fb98dead60b122495" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41315 } } peers { permanent_uuid: "4143869438d8444f9c6ade219e1a6d92" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34359 } }
I20251026 15:13:41.634433 1004 tablet_bootstrap.cc:654] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:41.634447 1000 ts_tablet_manager.cc:1434] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:13:41.634485 986 heartbeater.cc:499] Master 127.14.212.126:45387 was elected leader, sending a full tablet report...
I20251026 15:13:41.634382 1001 raft_consensus.cc:359] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46437 } } peers { permanent_uuid: "f5822b17117b4b2fb98dead60b122495" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41315 } } peers { permanent_uuid: "4143869438d8444f9c6ade219e1a6d92" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34359 } }
I20251026 15:13:41.634532 1001 raft_consensus.cc:385] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:41.634559 1001 raft_consensus.cc:740] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5822b17117b4b2fb98dead60b122495, State: Initialized, Role: FOLLOWER
I20251026 15:13:41.634644 1001 consensus_queue.cc:260] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46437 } } peers { permanent_uuid: "f5822b17117b4b2fb98dead60b122495" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41315 } } peers { permanent_uuid: "4143869438d8444f9c6ade219e1a6d92" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34359 } }
I20251026 15:13:41.634851 1004 log.cc:826] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:41.634866 1001 ts_tablet_manager.cc:1434] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:13:41.634935 724 heartbeater.cc:499] Master 127.14.212.126:45387 was elected leader, sending a full tablet report...
I20251026 15:13:41.635761 1004 tablet_bootstrap.cc:492] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92: No bootstrap required, opened a new log
I20251026 15:13:41.635814 1004 ts_tablet_manager.cc:1403] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:13:41.636860 1004 raft_consensus.cc:359] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46437 } } peers { permanent_uuid: "f5822b17117b4b2fb98dead60b122495" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41315 } } peers { permanent_uuid: "4143869438d8444f9c6ade219e1a6d92" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34359 } }
I20251026 15:13:41.636972 1004 raft_consensus.cc:385] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:41.636992 1004 raft_consensus.cc:740] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4143869438d8444f9c6ade219e1a6d92, State: Initialized, Role: FOLLOWER
I20251026 15:13:41.637060 1004 consensus_queue.cc:260] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46437 } } peers { permanent_uuid: "f5822b17117b4b2fb98dead60b122495" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41315 } } peers { permanent_uuid: "4143869438d8444f9c6ade219e1a6d92" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34359 } }
I20251026 15:13:41.637250 1004 ts_tablet_manager.cc:1434] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 15:13:41.637327 855 heartbeater.cc:499] Master 127.14.212.126:45387 was elected leader, sending a full tablet report...
I20251026 15:13:41.654834 1005 raft_consensus.cc:493] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:13:41.654958 1005 raft_consensus.cc:515] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46437 } } peers { permanent_uuid: "f5822b17117b4b2fb98dead60b122495" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41315 } } peers { permanent_uuid: "4143869438d8444f9c6ade219e1a6d92" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34359 } }
I20251026 15:13:41.655303 1005 leader_election.cc:290] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5822b17117b4b2fb98dead60b122495 (127.14.212.65:41315), 4143869438d8444f9c6ade219e1a6d92 (127.14.212.66:34359)
I20251026 15:13:41.658036 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.655471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:41315 (local address 127.14.212.67:54495)
1026 15:13:41.655546 (+ 75us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:41.655557 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:41.655601 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:41.655795 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:41.655796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:41.655799 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:41.656029 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.656031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.656989 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.656990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:41.657716 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.657718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.657797 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.657802 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:13:41.657836 (+ 34us) client_negotiation.cc:240] Negotiation successful
1026 15:13:41.657875 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20251026 15:13:41.658488 996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.655593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:54495 (local address 127.14.212.65:41315)
1026 15:13:41.655644 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:41.655647 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:41.655654 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:13:41.655684 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:41.655685 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:41.655696 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:41.655757 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:41.656133 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.656869 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.657816 (+ 947us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.658320 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.658330 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:41.658353 (+ 23us) server_negotiation.cc:300] Negotiation successful
1026 15:13:41.658382 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251026 15:13:41.658483 1009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.655537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:34359 (local address 127.14.212.67:58231)
1026 15:13:41.655714 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:41.655722 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:41.655777 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:41.655958 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:41.655959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:41.655960 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:41.656517 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.656520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.657382 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.657383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:41.658176 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:41.658178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.658289 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.658291 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:13:41.658327 (+ 36us) client_negotiation.cc:240] Negotiation successful
1026 15:13:41.658366 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":118,"threads_started":1}
I20251026 15:13:41.658785 678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c01a7fc0e62f42978e4bbe9b91a11ebc" candidate_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5822b17117b4b2fb98dead60b122495" is_pre_election: true
I20251026 15:13:41.658926 678 raft_consensus.cc:2468] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d383a6e623c1447e9aafeb1530ca5fb0 in term 0.
I20251026 15:13:41.658973 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.655616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:58231 (local address 127.14.212.66:34359)
1026 15:13:41.655663 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:41.655666 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:41.655780 (+ 114us) server_negotiation.cc:408] Connection header received
1026 15:13:41.655829 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:41.655830 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:41.655841 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:41.655912 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:41.656620 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.657265 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.658313 (+ 1048us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.658796 (+ 483us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.658804 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:41.658829 (+ 25us) server_negotiation.cc:300] Negotiation successful
1026 15:13:41.658851 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251026 15:13:41.659200 874 leader_election.cc:304] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d383a6e623c1447e9aafeb1530ca5fb0, f5822b17117b4b2fb98dead60b122495; no voters:
I20251026 15:13:41.659277 809 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c01a7fc0e62f42978e4bbe9b91a11ebc" candidate_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4143869438d8444f9c6ade219e1a6d92" is_pre_election: true
I20251026 15:13:41.659348 1005 raft_consensus.cc:2804] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:13:41.659412 1005 raft_consensus.cc:493] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:13:41.659436 1005 raft_consensus.cc:3060] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:41.659453 809 raft_consensus.cc:2468] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d383a6e623c1447e9aafeb1530ca5fb0 in term 0.
I20251026 15:13:41.660229 1005 raft_consensus.cc:515] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46437 } } peers { permanent_uuid: "f5822b17117b4b2fb98dead60b122495" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41315 } } peers { permanent_uuid: "4143869438d8444f9c6ade219e1a6d92" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34359 } }
I20251026 15:13:41.660364 1005 leader_election.cc:290] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [CANDIDATE]: Term 1 election: Requested vote from peers f5822b17117b4b2fb98dead60b122495 (127.14.212.65:41315), 4143869438d8444f9c6ade219e1a6d92 (127.14.212.66:34359)
I20251026 15:13:41.660548 678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c01a7fc0e62f42978e4bbe9b91a11ebc" candidate_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5822b17117b4b2fb98dead60b122495"
I20251026 15:13:41.660619 678 raft_consensus.cc:3060] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:41.661603 678 raft_consensus.cc:2468] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d383a6e623c1447e9aafeb1530ca5fb0 in term 1.
I20251026 15:13:41.661718 809 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c01a7fc0e62f42978e4bbe9b91a11ebc" candidate_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4143869438d8444f9c6ade219e1a6d92"
I20251026 15:13:41.661794 809 raft_consensus.cc:3060] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:41.661823 874 leader_election.cc:304] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d383a6e623c1447e9aafeb1530ca5fb0, f5822b17117b4b2fb98dead60b122495; no voters:
I20251026 15:13:41.661959 1005 raft_consensus.cc:2804] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:41.662202 1005 raft_consensus.cc:697] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [term 1 LEADER]: Becoming Leader. State: Replica: d383a6e623c1447e9aafeb1530ca5fb0, State: Running, Role: LEADER
I20251026 15:13:41.662328 1005 consensus_queue.cc:237] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46437 } } peers { permanent_uuid: "f5822b17117b4b2fb98dead60b122495" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41315 } } peers { permanent_uuid: "4143869438d8444f9c6ade219e1a6d92" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34359 } }
I20251026 15:13:41.662990 809 raft_consensus.cc:2468] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d383a6e623c1447e9aafeb1530ca5fb0 in term 1.
I20251026 15:13:41.663314 536 catalog_manager.cc:5649] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d383a6e623c1447e9aafeb1530ca5fb0 (127.14.212.67). New cstate: current_term: 1 leader_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d383a6e623c1447e9aafeb1530ca5fb0" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 46437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5822b17117b4b2fb98dead60b122495" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4143869438d8444f9c6ade219e1a6d92" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34359 } health_report { overall_health: UNKNOWN } } }
I20251026 15:13:41.673761 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.671852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.14.212.67:46437)
1026 15:13:41.671905 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:41.671906 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:41.671937 (+ 31us) server_negotiation.cc:408] Connection header received
1026 15:13:41.671971 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:41.671972 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:41.671989 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:41.672033 (+ 44us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:13:41.672249 (+ 216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.672748 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:41.673309 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:41.673423 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:41.673456 (+ 33us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:13:41.673545 (+ 89us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:13:41.673605 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:41.673653 (+ 48us) server_negotiation.cc:300] Negotiation successful
1026 15:13:41.673673 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251026 15:13:41.675488 809 raft_consensus.cc:1275] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92 [term 1 FOLLOWER]: Refusing update from remote peer d383a6e623c1447e9aafeb1530ca5fb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:41.675761 678 raft_consensus.cc:1275] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495 [term 1 FOLLOWER]: Refusing update from remote peer d383a6e623c1447e9aafeb1530ca5fb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:41.675866 1010 consensus_queue.cc:1048] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4143869438d8444f9c6ade219e1a6d92" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 34359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:41.676057 1010 consensus_queue.cc:1048] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5822b17117b4b2fb98dead60b122495" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 41315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:41.678530 1014 mvcc.cc:204] Tried to move back new op lower bound from 7215069682380382208 to 7215069682329726976. Current Snapshot: MvccSnapshot[applied={T|T < 7215069682380382208}]
I20251026 15:13:41.679301 1015 mvcc.cc:204] Tried to move back new op lower bound from 7215069682380382208 to 7215069682329726976. Current Snapshot: MvccSnapshot[applied={T|T < 7215069682380382208}]
I20251026 15:13:41.680684 534 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41062:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:41.680763 534 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41062:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:41.681429 534 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 52e5430d27b74e63b686a9ba8a50cdf4: Sending DeleteTablet for 3 replicas of tablet c01a7fc0e62f42978e4bbe9b91a11ebc
I20251026 15:13:41.681739 658 tablet_service.cc:1552] Processing DeleteTablet for tablet c01a7fc0e62f42978e4bbe9b91a11ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:41 UTC) from {username='slave'} at 127.0.0.1:41322
I20251026 15:13:41.681740 920 tablet_service.cc:1552] Processing DeleteTablet for tablet c01a7fc0e62f42978e4bbe9b91a11ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:41 UTC) from {username='slave'} at 127.0.0.1:56102
I20251026 15:13:41.681770 789 tablet_service.cc:1552] Processing DeleteTablet for tablet c01a7fc0e62f42978e4bbe9b91a11ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:41 UTC) from {username='slave'} at 127.0.0.1:58482
I20251026 15:13:41.681965 1022 tablet_replica.cc:333] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495: stopping tablet replica
I20251026 15:13:41.682054 1022 raft_consensus.cc:2243] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:41.682157 1022 raft_consensus.cc:2272] T c01a7fc0e62f42978e4bbe9b91a11ebc P f5822b17117b4b2fb98dead60b122495 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:41.682297 1023 tablet_replica.cc:333] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0: stopping tablet replica
I20251026 15:13:41.682297 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 596
I20251026 15:13:41.682765 1023 raft_consensus.cc:2243] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:13:41.683023 1023 raft_consensus.cc:2272] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:41.683504 1023 ts_tablet_manager.cc:1916] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:41.683719 1024 tablet_replica.cc:333] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92: stopping tablet replica
I20251026 15:13:41.683831 1024 raft_consensus.cc:2243] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:41.683969 1024 raft_consensus.cc:2272] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:41.684396 1024 ts_tablet_manager.cc:1916] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:41.685390 1023 ts_tablet_manager.cc:1929] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:41.685482 1023 log.cc:1199] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/c01a7fc0e62f42978e4bbe9b91a11ebc
I20251026 15:13:41.685761 1023 ts_tablet_manager.cc:1950] T c01a7fc0e62f42978e4bbe9b91a11ebc P d383a6e623c1447e9aafeb1530ca5fb0: Deleting consensus metadata
I20251026 15:13:41.686223 523 catalog_manager.cc:4985] TS d383a6e623c1447e9aafeb1530ca5fb0 (127.14.212.67:46437): tablet c01a7fc0e62f42978e4bbe9b91a11ebc (table test-table [id=4e8b0729156d47a58865a46082a73101]) successfully deleted
I20251026 15:13:41.686338 1024 ts_tablet_manager.cc:1929] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:41.686461 1024 log.cc:1199] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/c01a7fc0e62f42978e4bbe9b91a11ebc
I20251026 15:13:41.686717 1024 ts_tablet_manager.cc:1950] T c01a7fc0e62f42978e4bbe9b91a11ebc P 4143869438d8444f9c6ade219e1a6d92: Deleting consensus metadata
I20251026 15:13:41.687126 523 catalog_manager.cc:4985] TS 4143869438d8444f9c6ade219e1a6d92 (127.14.212.66:34359): tablet c01a7fc0e62f42978e4bbe9b91a11ebc (table test-table [id=4e8b0729156d47a58865a46082a73101]) successfully deleted
W20251026 15:13:41.689116 521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:41315 (error 108)
I20251026 15:13:41.689626 995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:41.689482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:41315 (local address 127.0.0.1:41330)
1026 15:13:41.689541 (+ 59us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:41.689588 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20251026 15:13:41.689651 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 727
W20251026 15:13:41.691283 521 catalog_manager.cc:4712] TS f5822b17117b4b2fb98dead60b122495 (127.14.212.65:41315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c01a7fc0e62f42978e4bbe9b91a11ebc: Network error: Client connection negotiation failed: client connection to 127.14.212.65:41315: connect: Connection refused (error 111)
I20251026 15:13:41.695062 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 858
I20251026 15:13:41.699937 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 505
2025-10-26T15:13:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (665 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:13:41 dist-test-slave-8pgz krb5kdc[1028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:13:41 dist-test-slave-8pgz krb5kdc[1028](info): set up 2 sockets
Oct 26 15:13:41 dist-test-slave-8pgz krb5kdc[1028](info): commencing operation
krb5kdc: starting...
W20251026 15:13:43.687634 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.968s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:13:43 dist-test-slave-8pgz krb5kdc[1028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:13:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:13:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:43.821939 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:41047
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:41047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:43.902700 1044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:43.902902 1044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:43.902926 1044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:43.904428 1044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:13:43.904479 1044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:43.904493 1044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:43.904507 1044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:13:43.904533 1044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:13:43.906261 1044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:41047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:41047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:43.906520 1044 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:43.906710 1044 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:43.909246 1050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:43.909278 1049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:43.909293 1052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:43.909504 1044 server_base.cc:1047] running on GCE node
I20251026 15:13:43.909705 1044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:43.909937 1044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:43.911088 1044 hybrid_clock.cc:648] HybridClock initialized: now 1761491623911063 us; error 36 us; skew 500 ppm
Oct 26 15:13:43 dist-test-slave-8pgz krb5kdc[1028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491623, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:43.913579 1044 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:13:43.914124 1044 webserver.cc:492] Webserver started at http://127.14.212.126:43155/ using document root <none> and password file <none>
I20251026 15:13:43.914315 1044 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:43.914361 1044 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:43.914516 1044 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:43.915486 1044 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "0699db8fd32241778fa4524a9f359c40"
format_stamp: "Formatted at 2025-10-26 15:13:43 on dist-test-slave-8pgz"
server_key: "de5226f4d769dbfba482a54babf675d2"
server_key_iv: "f41e9f742578171b98f03e4acbf38185"
server_key_version: "encryptionkey@0"
I20251026 15:13:43.915795 1044 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "0699db8fd32241778fa4524a9f359c40"
format_stamp: "Formatted at 2025-10-26 15:13:43 on dist-test-slave-8pgz"
server_key: "de5226f4d769dbfba482a54babf675d2"
server_key_iv: "f41e9f742578171b98f03e4acbf38185"
server_key_version: "encryptionkey@0"
I20251026 15:13:43.916993 1044 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:13:43.917641 1059 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:43.917814 1044 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:13:43.917876 1044 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "0699db8fd32241778fa4524a9f359c40"
format_stamp: "Formatted at 2025-10-26 15:13:43 on dist-test-slave-8pgz"
server_key: "de5226f4d769dbfba482a54babf675d2"
server_key_iv: "f41e9f742578171b98f03e4acbf38185"
server_key_version: "encryptionkey@0"
I20251026 15:13:43.917973 1044 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:43.936208 1044 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:43.938366 1044 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:43.938588 1044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:43.943981 1044 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:41047
I20251026 15:13:43.944041 1111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:41047 every 8 connection(s)
I20251026 15:13:43.944355 1044 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:13:43.944924 1112 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:43.946656 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 1044
I20251026 15:13:43.946772 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:13:43.946981 15185 external_mini_cluster.cc:1468] Setting key f4780cdefd43f1d18ea88f6181dc5ff8
I20251026 15:13:43.947540 1112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40: Bootstrap starting.
I20251026 15:13:43.948148 1112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:43.948390 1112 log.cc:826] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:43.949106 1112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40: No bootstrap required, opened a new log
I20251026 15:13:43.950117 1112 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0699db8fd32241778fa4524a9f359c40" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41047 } }
I20251026 15:13:43.950234 1112 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:43.950268 1112 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0699db8fd32241778fa4524a9f359c40, State: Initialized, Role: FOLLOWER
I20251026 15:13:43.950423 1112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0699db8fd32241778fa4524a9f359c40" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41047 } }
I20251026 15:13:43.950497 1112 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:13:43.950532 1112 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:13:43.950582 1112 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:43.951146 1112 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0699db8fd32241778fa4524a9f359c40" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41047 } }
I20251026 15:13:43.951264 1112 leader_election.cc:304] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0699db8fd32241778fa4524a9f359c40; no voters:
I20251026 15:13:43.951416 1112 leader_election.cc:290] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:13:43.951470 1117 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:43.951627 1117 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [term 1 LEADER]: Becoming Leader. State: Replica: 0699db8fd32241778fa4524a9f359c40, State: Running, Role: LEADER
I20251026 15:13:43.951653 1112 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [sys.catalog]: configured and running, proceeding with master startup.
Oct 26 15:13:43 dist-test-slave-8pgz krb5kdc[1028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:43.953320 1117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0699db8fd32241778fa4524a9f359c40" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41047 } }
I20251026 15:13:43.954315 1119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0699db8fd32241778fa4524a9f359c40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0699db8fd32241778fa4524a9f359c40" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41047 } } }
I20251026 15:13:43.954447 1119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:43.954306 1118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0699db8fd32241778fa4524a9f359c40. Latest consensus state: current_term: 1 leader_uuid: "0699db8fd32241778fa4524a9f359c40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0699db8fd32241778fa4524a9f359c40" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 41047 } } }
I20251026 15:13:43.954792 1118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40 [sys.catalog]: This master's current role is: LEADER
W20251026 15:13:43.954986 1131 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:13:43.955034 1131 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:13:43.955096 1132 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:13:43.955263 1132 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:13:43.956786 1132 catalog_manager.cc:1357] Generated new cluster ID: 653baf29f69742f3b7ee8f52e43e6a24
I20251026 15:13:43.956954 1132 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:13:43.957829 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:43.947520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47334 (local address 127.14.212.126:41047)
1026 15:13:43.947704 (+ 184us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:43.947717 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:43.947740 (+ 23us) server_negotiation.cc:408] Connection header received
1026 15:13:43.948092 (+ 352us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:43.948098 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:43.948157 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:43.948293 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:43.949501 (+ 1208us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:43.950227 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:43.950892 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:43.951151 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:43.953922 (+ 2771us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:43.953929 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:43.953929 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:43.953946 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:43.956230 (+ 2284us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:43.956727 (+ 497us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:43.956729 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:43.956730 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:43.956792 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:43.956992 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:43.956993 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:43.956994 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:43.957258 (+ 264us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:43.957386 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:43.957532 (+ 146us) server_negotiation.cc:300] Negotiation successful
1026 15:13:43.957652 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20251026 15:13:43.972548 1132 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:13:43.973120 1132 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:13:43.980306 1132 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40: Generated new TSK 0
I20251026 15:13:43.980556 1132 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:44.030529 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41047
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:44.109889 1140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:44.110087 1140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:44.110113 1140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:44.111666 1140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:44.111718 1140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:44.111747 1140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:13:44.113359 1140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:41047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:44.113569 1140 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:44.113758 1140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:44.116446 1145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:44.116420 1146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:44.116438 1140 server_base.cc:1047] running on GCE node
W20251026 15:13:44.116425 1148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:44.116756 1140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:44.116922 1140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:44.118052 1140 hybrid_clock.cc:648] HybridClock initialized: now 1761491624118035 us; error 30 us; skew 500 ppm
Oct 26 15:13:44 dist-test-slave-8pgz krb5kdc[1028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491624, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:44.120534 1140 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:13:44.121037 1140 webserver.cc:492] Webserver started at http://127.14.212.65:43219/ using document root <none> and password file <none>
I20251026 15:13:44.121222 1140 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:44.121263 1140 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:44.121373 1140 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:44.122364 1140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "daebccee809b4fcca1f98733701e1c2b"
format_stamp: "Formatted at 2025-10-26 15:13:44 on dist-test-slave-8pgz"
server_key: "8b2d147f7c9cc8aecf006466991c4cac"
server_key_iv: "aa3862d454944d389a7d2afb36b1bc45"
server_key_version: "encryptionkey@0"
I20251026 15:13:44.122713 1140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "daebccee809b4fcca1f98733701e1c2b"
format_stamp: "Formatted at 2025-10-26 15:13:44 on dist-test-slave-8pgz"
server_key: "8b2d147f7c9cc8aecf006466991c4cac"
server_key_iv: "aa3862d454944d389a7d2afb36b1bc45"
server_key_version: "encryptionkey@0"
I20251026 15:13:44.123872 1140 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:44.124631 1155 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:44.124801 1140 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:44.124876 1140 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "daebccee809b4fcca1f98733701e1c2b"
format_stamp: "Formatted at 2025-10-26 15:13:44 on dist-test-slave-8pgz"
server_key: "8b2d147f7c9cc8aecf006466991c4cac"
server_key_iv: "aa3862d454944d389a7d2afb36b1bc45"
server_key_version: "encryptionkey@0"
I20251026 15:13:44.124934 1140 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:44.147316 1140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:44.149408 1140 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:44.149551 1140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:44.149780 1140 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:44.150091 1140 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:44.150127 1140 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:44.150162 1140 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:44.150189 1140 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:44.155964 1140 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:38019
I20251026 15:13:44.156018 1268 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:38019 every 8 connection(s)
I20251026 15:13:44.156311 1140 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
Oct 26 15:13:44 dist-test-slave-8pgz krb5kdc[1028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491624, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:44.164824 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.156998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:56341 (local address 127.14.212.126:41047)
1026 15:13:44.157051 (+ 53us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:44.157052 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:44.157257 (+ 205us) server_negotiation.cc:408] Connection header received
1026 15:13:44.157779 (+ 522us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:44.157781 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:44.157793 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:44.157852 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:44.158664 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.159183 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.159735 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.159906 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.162171 (+ 2265us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:44.162175 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:44.162175 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:44.162182 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:44.163608 (+ 1426us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:44.164000 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:44.164004 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:44.164004 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:44.164028 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:44.164218 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:44.164220 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:44.164220 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:44.164317 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:44.164382 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:44.164628 (+ 246us) server_negotiation.cc:300] Negotiation successful
1026 15:13:44.164730 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251026 15:13:44.164934 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.156985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:41047 (local address 127.14.212.65:56341)
1026 15:13:44.157137 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:44.157163 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:44.157674 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:44.157912 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:44.157915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:44.158224 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:44.158576 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.158580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.159268 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.159269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:44.159658 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.159660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.159790 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.160757 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:44.160772 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:44.162081 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:44.163691 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:44.163697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:44.163700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:44.163925 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:44.164084 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:44.164085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:44.164085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:44.164163 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:44.164376 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:44.164381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:44.164508 (+ 127us) client_negotiation.cc:769] Sending connection context
1026 15:13:44.164603 (+ 95us) client_negotiation.cc:240] Negotiation successful
1026 15:13:44.164737 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20251026 15:13:44.165479 1269 heartbeater.cc:344] Connected to a master server at 127.14.212.126:41047
I20251026 15:13:44.165565 1269 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:44.165546 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 1140
I20251026 15:13:44.165673 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:13:44.165800 1269 heartbeater.cc:507] Master 127.14.212.126:41047 requested a full tablet report, sending...
I20251026 15:13:44.165864 15185 external_mini_cluster.cc:1468] Setting key a1073e5556b6e284e52a4e4cb3366686
I20251026 15:13:44.166292 1076 ts_manager.cc:194] Registered new tserver with Master: daebccee809b4fcca1f98733701e1c2b (127.14.212.65:38019)
I20251026 15:13:44.167088 1076 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:56341
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:44.211012 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41047
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:44.293058 1276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:44.293249 1276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:44.293277 1276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:44.294741 1276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:44.294791 1276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:44.294819 1276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:13:44.296773 1276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:41047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:44.297027 1276 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:44.297194 1276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:44.299886 1284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:44.299981 1281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:44.300020 1276 server_base.cc:1047] running on GCE node
W20251026 15:13:44.300102 1282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:44.300280 1276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:44.300498 1276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:44.301640 1276 hybrid_clock.cc:648] HybridClock initialized: now 1761491624301625 us; error 25 us; skew 500 ppm
Oct 26 15:13:44 dist-test-slave-8pgz krb5kdc[1028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491624, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:44.304116 1276 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:13:44.304633 1276 webserver.cc:492] Webserver started at http://127.14.212.66:40061/ using document root <none> and password file <none>
I20251026 15:13:44.304818 1276 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:44.304858 1276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:44.304982 1276 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:44.305954 1276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "18f123bb2e52449b9d4682b6f3c47f9f"
format_stamp: "Formatted at 2025-10-26 15:13:44 on dist-test-slave-8pgz"
server_key: "58625675a8b4b953d535107258fb8c4b"
server_key_iv: "f9b9844d34c0b982f43c01b9b325032f"
server_key_version: "encryptionkey@0"
I20251026 15:13:44.306265 1276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "18f123bb2e52449b9d4682b6f3c47f9f"
format_stamp: "Formatted at 2025-10-26 15:13:44 on dist-test-slave-8pgz"
server_key: "58625675a8b4b953d535107258fb8c4b"
server_key_iv: "f9b9844d34c0b982f43c01b9b325032f"
server_key_version: "encryptionkey@0"
I20251026 15:13:44.307548 1276 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:44.308338 1291 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:44.308554 1276 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:44.308636 1276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "18f123bb2e52449b9d4682b6f3c47f9f"
format_stamp: "Formatted at 2025-10-26 15:13:44 on dist-test-slave-8pgz"
server_key: "58625675a8b4b953d535107258fb8c4b"
server_key_iv: "f9b9844d34c0b982f43c01b9b325032f"
server_key_version: "encryptionkey@0"
I20251026 15:13:44.308702 1276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:44.325250 1276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:44.327252 1276 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:44.327396 1276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:44.327611 1276 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:44.327926 1276 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:44.327960 1276 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:44.327998 1276 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:44.328029 1276 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:44.333616 1276 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:35123
I20251026 15:13:44.333680 1404 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:35123 every 8 connection(s)
I20251026 15:13:44.333986 1276 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:13:44.335707 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 1276
I20251026 15:13:44.335824 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:13:44.336014 15185 external_mini_cluster.cc:1468] Setting key 72487c5f829e9379ff1f3a5872d1a661
Oct 26 15:13:44 dist-test-slave-8pgz krb5kdc[1028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491624, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:44.343990 1407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.334682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:41047 (local address 127.14.212.66:54575)
1026 15:13:44.334831 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:44.334858 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:44.335339 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:44.335541 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:44.335546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:44.336066 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:44.336441 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.336445 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.337241 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.337243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:44.337874 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.337876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.338031 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.339092 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:44.339108 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:44.340784 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:44.342814 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:44.342820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:44.342822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:44.343032 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:44.343210 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:44.343210 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:44.343211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:44.343263 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:44.343520 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:44.343522 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:44.343638 (+ 116us) client_negotiation.cc:769] Sending connection context
1026 15:13:44.343719 (+ 81us) client_negotiation.cc:240] Negotiation successful
1026 15:13:44.343828 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20251026 15:13:44.344002 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.334729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:54575 (local address 127.14.212.126:41047)
1026 15:13:44.334792 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:44.334794 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:44.334942 (+ 148us) server_negotiation.cc:408] Connection header received
1026 15:13:44.335414 (+ 472us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:44.335415 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:44.335424 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:44.335477 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:44.336529 (+ 1052us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.337129 (+ 600us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.337995 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.338127 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.340892 (+ 2765us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:44.340896 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:44.340897 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:44.340907 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:44.342728 (+ 1821us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:44.343113 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:44.343114 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:44.343114 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:44.343142 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:44.343325 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:44.343326 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:44.343327 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:44.343461 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:44.343539 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:44.343765 (+ 226us) server_negotiation.cc:300] Negotiation successful
1026 15:13:44.343882 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
I20251026 15:13:44.344440 1405 heartbeater.cc:344] Connected to a master server at 127.14.212.126:41047
I20251026 15:13:44.344535 1405 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:44.344733 1405 heartbeater.cc:507] Master 127.14.212.126:41047 requested a full tablet report, sending...
I20251026 15:13:44.345093 1076 ts_manager.cc:194] Registered new tserver with Master: 18f123bb2e52449b9d4682b6f3c47f9f (127.14.212.66:35123)
I20251026 15:13:44.345503 1076 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:54575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:44.382725 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:41047
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:44.462059 1412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:44.462255 1412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:44.462278 1412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:44.463778 1412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:44.463833 1412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:44.463860 1412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:13:44.465505 1412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:41047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:44.465732 1412 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:44.465934 1412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:44.468734 1417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:44.468737 1418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:44.468772 1420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:44.469099 1412 server_base.cc:1047] running on GCE node
I20251026 15:13:44.469247 1412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:44.469430 1412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:44.470566 1412 hybrid_clock.cc:648] HybridClock initialized: now 1761491624470550 us; error 32 us; skew 500 ppm
Oct 26 15:13:44 dist-test-slave-8pgz krb5kdc[1028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491624, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:44.473084 1412 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:13:44.473596 1412 webserver.cc:492] Webserver started at http://127.14.212.67:36469/ using document root <none> and password file <none>
I20251026 15:13:44.473778 1412 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:44.473816 1412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:44.473937 1412 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:44.474922 1412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "617d6fca0f634864a480c56557a0be89"
format_stamp: "Formatted at 2025-10-26 15:13:44 on dist-test-slave-8pgz"
server_key: "72df1fc5fe8fa93e52805f8ff4aacd72"
server_key_iv: "26b98db6a8f675ac4c0df00953f37adc"
server_key_version: "encryptionkey@0"
I20251026 15:13:44.475229 1412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "617d6fca0f634864a480c56557a0be89"
format_stamp: "Formatted at 2025-10-26 15:13:44 on dist-test-slave-8pgz"
server_key: "72df1fc5fe8fa93e52805f8ff4aacd72"
server_key_iv: "26b98db6a8f675ac4c0df00953f37adc"
server_key_version: "encryptionkey@0"
I20251026 15:13:44.476467 1412 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:13:44.477207 1427 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:44.477427 1412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:44.477515 1412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "617d6fca0f634864a480c56557a0be89"
format_stamp: "Formatted at 2025-10-26 15:13:44 on dist-test-slave-8pgz"
server_key: "72df1fc5fe8fa93e52805f8ff4aacd72"
server_key_iv: "26b98db6a8f675ac4c0df00953f37adc"
server_key_version: "encryptionkey@0"
I20251026 15:13:44.477587 1412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:44.506438 1412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:44.508373 1412 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:44.508509 1412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:44.508752 1412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:44.509081 1412 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:44.509116 1412 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:44.509189 1412 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:44.509213 1412 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:44.514917 1412 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:45537
I20251026 15:13:44.514971 1540 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:45537 every 8 connection(s)
I20251026 15:13:44.515275 1412 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:13:44.517143 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 1412
I20251026 15:13:44.517239 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:13:44.517407 15185 external_mini_cluster.cc:1468] Setting key 58f535efd4a5831478aa75a5de80e758
Oct 26 15:13:44 dist-test-slave-8pgz krb5kdc[1028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491624, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:44.523736 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.515988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:53559 (local address 127.14.212.126:41047)
1026 15:13:44.516057 (+ 69us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:44.516059 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:44.516202 (+ 143us) server_negotiation.cc:408] Connection header received
1026 15:13:44.516674 (+ 472us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:44.516675 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:44.516684 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:44.516736 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:44.517580 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.518162 (+ 582us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.518789 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.518913 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.521234 (+ 2321us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:44.521237 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:44.521238 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:44.521244 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:44.522569 (+ 1325us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:44.522926 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:44.522927 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:44.522928 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:44.522954 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:44.523123 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:44.523124 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:44.523125 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:44.523217 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:44.523286 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:44.523535 (+ 249us) server_negotiation.cc:300] Negotiation successful
1026 15:13:44.523630 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251026 15:13:44.523833 1543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.515913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:41047 (local address 127.14.212.67:53559)
1026 15:13:44.516099 (+ 186us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:44.516126 (+ 27us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:44.516606 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:44.516796 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:44.516800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:44.517150 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:44.517483 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.517487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.518261 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.518263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:44.518722 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.518724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.518830 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.519735 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:44.519750 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:44.521132 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:44.522647 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:44.522653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:44.522656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:44.522859 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:44.523010 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:44.523011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:44.523011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:44.523068 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:44.523284 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:44.523290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:44.523414 (+ 124us) client_negotiation.cc:769] Sending connection context
1026 15:13:44.523511 (+ 97us) client_negotiation.cc:240] Negotiation successful
1026 15:13:44.523643 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20251026 15:13:44.524307 1541 heartbeater.cc:344] Connected to a master server at 127.14.212.126:41047
I20251026 15:13:44.524408 1541 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:44.524602 1541 heartbeater.cc:507] Master 127.14.212.126:41047 requested a full tablet report, sending...
I20251026 15:13:44.524940 1076 ts_manager.cc:194] Registered new tserver with Master: 617d6fca0f634864a480c56557a0be89 (127.14.212.67:45537)
I20251026 15:13:44.525384 1076 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:53559
I20251026 15:13:44.529521 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:13:44.536196 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.530893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.14.212.126:41047)
1026 15:13:44.530933 (+ 40us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:44.530935 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:44.531021 (+ 86us) server_negotiation.cc:408] Connection header received
1026 15:13:44.531069 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:44.531070 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:44.531079 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:44.531125 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:44.531797 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.532267 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.532757 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.532858 (+ 101us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.534006 (+ 1148us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:44.534009 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:44.534009 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:44.534014 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:44.535287 (+ 1273us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:44.535636 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:44.535636 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:44.535637 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:44.535654 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:44.535787 (+ 133us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:44.535787 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:44.535787 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:44.535878 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:44.535959 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:44.536052 (+ 93us) server_negotiation.cc:300] Negotiation successful
1026 15:13:44.536096 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20251026 15:13:44.537918 1076 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:13:44.539258 1076 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:13:44.544481 1552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.541450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:38019 (local address 127.0.0.1:33648)
1026 15:13:44.541922 (+ 472us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:44.541930 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:44.542005 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:44.542238 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:44.542239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:44.542241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:44.542484 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.542487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.543376 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.543377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:44.544188 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.544190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.544267 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.544308 (+ 41us) client_negotiation.cc:769] Sending connection context
1026 15:13:44.544357 (+ 49us) client_negotiation.cc:240] Negotiation successful
1026 15:13:44.544404 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":82,"threads_started":1}
I20251026 15:13:44.544718 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.541446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:35123 (local address 127.0.0.1:46950)
1026 15:13:44.541809 (+ 363us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:44.541818 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:44.541860 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:44.542158 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:44.542159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:44.542160 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:44.542338 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.542339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.543639 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.543639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:44.544480 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.544482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.544573 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.544578 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:13:44.544617 (+ 39us) client_negotiation.cc:240] Negotiation successful
1026 15:13:44.544655 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":110,"threads_started":1}
I20251026 15:13:44.545099 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.541702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.14.212.65:38019)
1026 15:13:44.541849 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:44.541858 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:44.541994 (+ 136us) server_negotiation.cc:408] Connection header received
1026 15:13:44.542079 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:44.542081 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:44.542099 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:44.542161 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:44.542579 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.543295 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.544299 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.544887 (+ 588us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.544947 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:44.544978 (+ 31us) server_negotiation.cc:300] Negotiation successful
1026 15:13:44.545021 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20251026 15:13:44.545380 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.541839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.14.212.66:35123)
1026 15:13:44.541969 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:44.541978 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:44.541989 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:13:44.542028 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:44.542030 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:44.542049 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:44.542145 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:44.542453 (+ 308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.543553 (+ 1100us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.544660 (+ 1107us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.545168 (+ 508us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.545221 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:44.545251 (+ 30us) server_negotiation.cc:300] Negotiation successful
1026 15:13:44.545294 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20251026 15:13:44.545782 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.541440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:45537 (local address 127.0.0.1:60868)
1026 15:13:44.541619 (+ 179us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:44.541645 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:44.541719 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:44.541985 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:44.541987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:44.541989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:44.542275 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.542277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.544770 (+ 2493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.544771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:44.545565 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.545567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.545641 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.545645 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:44.545675 (+ 30us) client_negotiation.cc:240] Negotiation successful
1026 15:13:44.545711 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20251026 15:13:44.545744 1203 tablet_service.cc:1505] Processing CreateTablet for tablet 61e32e000f9a4f6597236db89f69f4d9 (DEFAULT_TABLE table=test-table [id=3db4a489bbc44981b5e96af2533b9116]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:44.546069 1203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e32e000f9a4f6597236db89f69f4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:44.546305 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.541537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.14.212.67:45537)
1026 15:13:44.541728 (+ 191us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:44.541737 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:44.541750 (+ 13us) server_negotiation.cc:408] Connection header received
1026 15:13:44.541792 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:44.541795 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:44.541813 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:44.541881 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:44.542362 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.543548 (+ 1186us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.545672 (+ 2124us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.546130 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.546182 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:44.546208 (+ 26us) server_negotiation.cc:300] Negotiation successful
1026 15:13:44.546251 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20251026 15:13:44.546913 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 61e32e000f9a4f6597236db89f69f4d9 (DEFAULT_TABLE table=test-table [id=3db4a489bbc44981b5e96af2533b9116]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:44.547206 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e32e000f9a4f6597236db89f69f4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:44.548535 1558 tablet_bootstrap.cc:492] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b: Bootstrap starting.
I20251026 15:13:44.548921 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 61e32e000f9a4f6597236db89f69f4d9 (DEFAULT_TABLE table=test-table [id=3db4a489bbc44981b5e96af2533b9116]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:44.549173 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e32e000f9a4f6597236db89f69f4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:44.549230 1558 tablet_bootstrap.cc:654] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:44.549654 1559 tablet_bootstrap.cc:492] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89: Bootstrap starting.
I20251026 15:13:44.549587 1558 log.cc:826] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:44.550318 1559 tablet_bootstrap.cc:654] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:44.550648 1559 log.cc:826] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:44.550710 1558 tablet_bootstrap.cc:492] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b: No bootstrap required, opened a new log
I20251026 15:13:44.550769 1558 ts_tablet_manager.cc:1403] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:13:44.551666 1562 tablet_bootstrap.cc:492] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f: Bootstrap starting.
I20251026 15:13:44.552261 1559 tablet_bootstrap.cc:492] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89: No bootstrap required, opened a new log
I20251026 15:13:44.552271 1562 tablet_bootstrap.cc:654] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:44.552416 1559 ts_tablet_manager.cc:1403] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251026 15:13:44.552361 1558 raft_consensus.cc:359] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617d6fca0f634864a480c56557a0be89" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45537 } } peers { permanent_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35123 } } peers { permanent_uuid: "daebccee809b4fcca1f98733701e1c2b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38019 } }
I20251026 15:13:44.552546 1558 raft_consensus.cc:385] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:44.552608 1558 raft_consensus.cc:740] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daebccee809b4fcca1f98733701e1c2b, State: Initialized, Role: FOLLOWER
I20251026 15:13:44.552628 1562 log.cc:826] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:44.552733 1558 consensus_queue.cc:260] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617d6fca0f634864a480c56557a0be89" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45537 } } peers { permanent_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35123 } } peers { permanent_uuid: "daebccee809b4fcca1f98733701e1c2b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38019 } }
I20251026 15:13:44.553053 1558 ts_tablet_manager.cc:1434] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:13:44.553447 1562 tablet_bootstrap.cc:492] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f: No bootstrap required, opened a new log
I20251026 15:13:44.553520 1562 ts_tablet_manager.cc:1403] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:13:44.554170 1559 raft_consensus.cc:359] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617d6fca0f634864a480c56557a0be89" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45537 } } peers { permanent_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35123 } } peers { permanent_uuid: "daebccee809b4fcca1f98733701e1c2b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38019 } }
I20251026 15:13:44.554291 1559 raft_consensus.cc:385] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:44.554324 1559 raft_consensus.cc:740] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617d6fca0f634864a480c56557a0be89, State: Initialized, Role: FOLLOWER
I20251026 15:13:44.554459 1559 consensus_queue.cc:260] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617d6fca0f634864a480c56557a0be89" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45537 } } peers { permanent_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35123 } } peers { permanent_uuid: "daebccee809b4fcca1f98733701e1c2b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38019 } }
I20251026 15:13:44.554734 1559 ts_tablet_manager.cc:1434] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:13:44.554764 1541 heartbeater.cc:499] Master 127.14.212.126:41047 was elected leader, sending a full tablet report...
I20251026 15:13:44.555063 1562 raft_consensus.cc:359] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617d6fca0f634864a480c56557a0be89" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45537 } } peers { permanent_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35123 } } peers { permanent_uuid: "daebccee809b4fcca1f98733701e1c2b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38019 } }
I20251026 15:13:44.555173 1562 raft_consensus.cc:385] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:44.555194 1562 raft_consensus.cc:740] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18f123bb2e52449b9d4682b6f3c47f9f, State: Initialized, Role: FOLLOWER
I20251026 15:13:44.555276 1562 consensus_queue.cc:260] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617d6fca0f634864a480c56557a0be89" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45537 } } peers { permanent_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35123 } } peers { permanent_uuid: "daebccee809b4fcca1f98733701e1c2b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38019 } }
I20251026 15:13:44.555516 1562 ts_tablet_manager.cc:1434] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:13:44.553095 1269 heartbeater.cc:499] Master 127.14.212.126:41047 was elected leader, sending a full tablet report...
I20251026 15:13:44.555593 1405 heartbeater.cc:499] Master 127.14.212.126:41047 was elected leader, sending a full tablet report...
W20251026 15:13:44.584550 1406 tablet.cc:2378] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 15:13:44.657037 1270 tablet.cc:2378] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:13:44.686810 1566 raft_consensus.cc:493] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:13:44.686939 1566 raft_consensus.cc:515] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617d6fca0f634864a480c56557a0be89" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45537 } } peers { permanent_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35123 } } peers { permanent_uuid: "daebccee809b4fcca1f98733701e1c2b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38019 } }
I20251026 15:13:44.687311 1566 leader_election.cc:290] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 617d6fca0f634864a480c56557a0be89 (127.14.212.67:45537), daebccee809b4fcca1f98733701e1c2b (127.14.212.65:38019)
I20251026 15:13:44.689679 1407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.687505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:45537 (local address 127.14.212.66:36631)
1026 15:13:44.687568 (+ 63us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:44.687581 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:44.687629 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:44.687795 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:44.687796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:44.687798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:44.688048 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.688050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.688859 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.688860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:44.689454 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.689456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.689527 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.689532 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:13:44.689566 (+ 34us) client_negotiation.cc:240] Negotiation successful
1026 15:13:44.689606 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20251026 15:13:44.690106 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.687581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:36631 (local address 127.14.212.67:45537)
1026 15:13:44.687645 (+ 64us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:44.687647 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:44.687658 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:13:44.687692 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:44.687693 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:44.687705 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:44.687781 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:44.688130 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.688775 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.689543 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.690006 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.690015 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:44.690035 (+ 20us) server_negotiation.cc:300] Negotiation successful
1026 15:13:44.690060 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251026 15:13:44.690440 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.687523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:38019 (local address 127.14.212.66:37423)
1026 15:13:44.687727 (+ 204us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:44.687737 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:44.687800 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:44.688004 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:44.688005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:44.688007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:44.688304 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.688307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.689059 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.689060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:44.689638 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:44.689640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.690228 (+ 588us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.690232 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:44.690288 (+ 56us) client_negotiation.cc:240] Negotiation successful
1026 15:13:44.690332 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":131,"threads_started":1}
I20251026 15:13:44.690429 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.687609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:37423 (local address 127.14.212.65:38019)
1026 15:13:44.687667 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:44.687669 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:44.687801 (+ 132us) server_negotiation.cc:408] Connection header received
1026 15:13:44.687866 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:44.687867 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:44.687878 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:44.687941 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:44.688400 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.688954 (+ 554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.689766 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.690184 (+ 418us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.690194 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:44.690311 (+ 117us) server_negotiation.cc:300] Negotiation successful
1026 15:13:44.690349 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251026 15:13:44.690426 1495 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61e32e000f9a4f6597236db89f69f4d9" candidate_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617d6fca0f634864a480c56557a0be89" is_pre_election: true
I20251026 15:13:44.690593 1495 raft_consensus.cc:2468] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18f123bb2e52449b9d4682b6f3c47f9f in term 0.
I20251026 15:13:44.690845 1295 leader_election.cc:304] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18f123bb2e52449b9d4682b6f3c47f9f, 617d6fca0f634864a480c56557a0be89; no voters:
I20251026 15:13:44.690935 1223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61e32e000f9a4f6597236db89f69f4d9" candidate_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daebccee809b4fcca1f98733701e1c2b" is_pre_election: true
I20251026 15:13:44.690989 1566 raft_consensus.cc:2804] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:13:44.691043 1566 raft_consensus.cc:493] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:13:44.691073 1566 raft_consensus.cc:3060] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:44.691082 1223 raft_consensus.cc:2468] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18f123bb2e52449b9d4682b6f3c47f9f in term 0.
I20251026 15:13:44.691764 1566 raft_consensus.cc:515] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617d6fca0f634864a480c56557a0be89" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45537 } } peers { permanent_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35123 } } peers { permanent_uuid: "daebccee809b4fcca1f98733701e1c2b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38019 } }
I20251026 15:13:44.691919 1566 leader_election.cc:290] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [CANDIDATE]: Term 1 election: Requested vote from peers 617d6fca0f634864a480c56557a0be89 (127.14.212.67:45537), daebccee809b4fcca1f98733701e1c2b (127.14.212.65:38019)
I20251026 15:13:44.692075 1495 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61e32e000f9a4f6597236db89f69f4d9" candidate_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617d6fca0f634864a480c56557a0be89"
I20251026 15:13:44.692090 1223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61e32e000f9a4f6597236db89f69f4d9" candidate_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daebccee809b4fcca1f98733701e1c2b"
I20251026 15:13:44.692147 1223 raft_consensus.cc:3060] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:44.692149 1495 raft_consensus.cc:3060] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:44.692806 1495 raft_consensus.cc:2468] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18f123bb2e52449b9d4682b6f3c47f9f in term 1.
I20251026 15:13:44.692819 1223 raft_consensus.cc:2468] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18f123bb2e52449b9d4682b6f3c47f9f in term 1.
I20251026 15:13:44.692958 1295 leader_election.cc:304] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18f123bb2e52449b9d4682b6f3c47f9f, 617d6fca0f634864a480c56557a0be89; no voters:
I20251026 15:13:44.693065 1566 raft_consensus.cc:2804] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:44.693228 1566 raft_consensus.cc:697] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [term 1 LEADER]: Becoming Leader. State: Replica: 18f123bb2e52449b9d4682b6f3c47f9f, State: Running, Role: LEADER
I20251026 15:13:44.693332 1566 consensus_queue.cc:237] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617d6fca0f634864a480c56557a0be89" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45537 } } peers { permanent_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35123 } } peers { permanent_uuid: "daebccee809b4fcca1f98733701e1c2b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38019 } }
I20251026 15:13:44.694139 1075 catalog_manager.cc:5649] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 18f123bb2e52449b9d4682b6f3c47f9f (127.14.212.66). New cstate: current_term: 1 leader_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617d6fca0f634864a480c56557a0be89" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18f123bb2e52449b9d4682b6f3c47f9f" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 35123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "daebccee809b4fcca1f98733701e1c2b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38019 } health_report { overall_health: UNKNOWN } } }
I20251026 15:13:44.729252 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.726751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.14.212.66:35123)
1026 15:13:44.726811 (+ 60us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:44.726813 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:44.726821 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:13:44.726866 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:44.726867 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:44.726912 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:44.726968 (+ 56us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:13:44.727379 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.727882 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.728499 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.728673 (+ 174us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.728846 (+ 173us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:13:44.728957 (+ 111us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:13:44.729060 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:44.729166 (+ 106us) server_negotiation.cc:300] Negotiation successful
1026 15:13:44.729179 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251026 15:13:44.731019 1495 raft_consensus.cc:1275] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89 [term 1 FOLLOWER]: Refusing update from remote peer 18f123bb2e52449b9d4682b6f3c47f9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:44.731024 1223 raft_consensus.cc:1275] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b [term 1 FOLLOWER]: Refusing update from remote peer 18f123bb2e52449b9d4682b6f3c47f9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:44.731333 1568 consensus_queue.cc:1048] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "daebccee809b4fcca1f98733701e1c2b" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 38019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:44.731483 1566 consensus_queue.cc:1048] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "617d6fca0f634864a480c56557a0be89" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:44.733574 1571 mvcc.cc:204] Tried to move back new op lower bound from 7215069694895521792 to 7215069694744563712. Current Snapshot: MvccSnapshot[applied={T|T < 7215069694895521792}]
I20251026 15:13:44.733803 1572 mvcc.cc:204] Tried to move back new op lower bound from 7215069694895521792 to 7215069694744563712. Current Snapshot: MvccSnapshot[applied={T|T < 7215069694895521792}]
I20251026 15:13:44.734140 1573 mvcc.cc:204] Tried to move back new op lower bound from 7215069694895521792 to 7215069694744563712. Current Snapshot: MvccSnapshot[applied={T|T < 7215069694895521792}]
I20251026 15:13:44.736330 1076 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47350:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:44.736411 1076 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47350:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:44.737126 1076 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0699db8fd32241778fa4524a9f359c40: Sending DeleteTablet for 3 replicas of tablet 61e32e000f9a4f6597236db89f69f4d9
I20251026 15:13:44.737464 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 61e32e000f9a4f6597236db89f69f4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:44 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:46950
I20251026 15:13:44.737468 1203 tablet_service.cc:1552] Processing DeleteTablet for tablet 61e32e000f9a4f6597236db89f69f4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:44 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:33648
I20251026 15:13:44.737680 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 61e32e000f9a4f6597236db89f69f4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:44 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:60868
I20251026 15:13:44.737754 1582 tablet_replica.cc:333] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b: stopping tablet replica
I20251026 15:13:44.737839 1582 raft_consensus.cc:2243] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:44.737900 1583 tablet_replica.cc:333] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89: stopping tablet replica
I20251026 15:13:44.737941 1582 raft_consensus.cc:2272] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:44.738020 1583 raft_consensus.cc:2243] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:44.738127 1581 tablet_replica.cc:333] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f: stopping tablet replica
I20251026 15:13:44.738137 1583 raft_consensus.cc:2272] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:44.738255 1581 raft_consensus.cc:2243] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:13:44.738415 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 1140
I20251026 15:13:44.738682 1583 ts_tablet_manager.cc:1916] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:44.738683 1582 ts_tablet_manager.cc:1916] T 61e32e000f9a4f6597236db89f69f4d9 P daebccee809b4fcca1f98733701e1c2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:44.738991 1581 raft_consensus.cc:2272] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:44.739404 1581 ts_tablet_manager.cc:1916] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:44.741001 1581 ts_tablet_manager.cc:1929] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:44.741034 1583 ts_tablet_manager.cc:1929] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:44.741091 1581 log.cc:1199] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/61e32e000f9a4f6597236db89f69f4d9
I20251026 15:13:44.741128 1583 log.cc:1199] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/61e32e000f9a4f6597236db89f69f4d9
I20251026 15:13:44.741407 1581 ts_tablet_manager.cc:1950] T 61e32e000f9a4f6597236db89f69f4d9 P 18f123bb2e52449b9d4682b6f3c47f9f: Deleting consensus metadata
I20251026 15:13:44.741442 1583 ts_tablet_manager.cc:1950] T 61e32e000f9a4f6597236db89f69f4d9 P 617d6fca0f634864a480c56557a0be89: Deleting consensus metadata
I20251026 15:13:44.741824 1062 catalog_manager.cc:4985] TS 18f123bb2e52449b9d4682b6f3c47f9f (127.14.212.66:35123): tablet 61e32e000f9a4f6597236db89f69f4d9 (table test-table [id=3db4a489bbc44981b5e96af2533b9116]) successfully deleted
I20251026 15:13:44.742132 1063 catalog_manager.cc:4985] TS 617d6fca0f634864a480c56557a0be89 (127.14.212.67:45537): tablet 61e32e000f9a4f6597236db89f69f4d9 (table test-table [id=3db4a489bbc44981b5e96af2533b9116]) successfully deleted
W20251026 15:13:44.745124 1061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:38019 (error 108)
I20251026 15:13:44.745637 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 1276
I20251026 15:13:44.746430 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.745735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:38019 (local address 127.0.0.1:33664)
1026 15:13:44.746336 (+ 601us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:44.746376 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580}
W20251026 15:13:44.746549 1061 catalog_manager.cc:4712] TS daebccee809b4fcca1f98733701e1c2b (127.14.212.65:38019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61e32e000f9a4f6597236db89f69f4d9: Network error: Client connection negotiation failed: client connection to 127.14.212.65:38019: connect: Connection refused (error 111)
I20251026 15:13:44.751116 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 1412
I20251026 15:13:44.755632 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 1044
2025-10-26T15:13:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3056 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2025-10-26T15:13:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:13:44Z Disabled control of system clock
I20251026 15:13:44.782709 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:45351
--webserver_interface=127.14.212.126
--webserver_port=0
--builtin_ntp_servers=127.14.212.84:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:45351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251026 15:13:44.862329 1588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:44.862565 1588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:44.862598 1588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:44.864078 1588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:13:44.864125 1588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:44.864143 1588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:44.864163 1588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:13:44.864180 1588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:13:44.865710 1588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:45351
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:45351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:44.865979 1588 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:44.866210 1588 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:44.868741 1596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:44.868752 1593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:44.868810 1588 server_base.cc:1047] running on GCE node
W20251026 15:13:44.868953 1594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:44.869194 1588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:44.869414 1588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:44.870590 1588 hybrid_clock.cc:648] HybridClock initialized: now 1761491624870573 us; error 28 us; skew 500 ppm
I20251026 15:13:44.871735 1588 webserver.cc:492] Webserver started at http://127.14.212.126:34147/ using document root <none> and password file <none>
I20251026 15:13:44.871935 1588 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:44.871979 1588 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:44.872100 1588 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:44.873051 1588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "220b5b30188d42b6983502b5bd18bd93"
format_stamp: "Formatted at 2025-10-26 15:13:44 on dist-test-slave-8pgz"
server_key: "09bc1cdd7ea8340d9b227ab7653e1354"
server_key_iv: "5ef767af2401cb8a3175d3e2af5d9cb9"
server_key_version: "encryptionkey@0"
I20251026 15:13:44.873423 1588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "220b5b30188d42b6983502b5bd18bd93"
format_stamp: "Formatted at 2025-10-26 15:13:44 on dist-test-slave-8pgz"
server_key: "09bc1cdd7ea8340d9b227ab7653e1354"
server_key_iv: "5ef767af2401cb8a3175d3e2af5d9cb9"
server_key_version: "encryptionkey@0"
I20251026 15:13:44.875483 1588 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20251026 15:13:44.876355 1602 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:44.876561 1588 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:44.876654 1588 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "220b5b30188d42b6983502b5bd18bd93"
format_stamp: "Formatted at 2025-10-26 15:13:44 on dist-test-slave-8pgz"
server_key: "09bc1cdd7ea8340d9b227ab7653e1354"
server_key_iv: "5ef767af2401cb8a3175d3e2af5d9cb9"
server_key_version: "encryptionkey@0"
I20251026 15:13:44.876729 1588 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:44.900542 1588 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:44.900882 1588 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:44.901141 1588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:44.906203 1588 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:45351
I20251026 15:13:44.906248 1654 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:45351 every 8 connection(s)
I20251026 15:13:44.906641 1588 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:13:44.907435 1655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:44.911336 1655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93: Bootstrap starting.
I20251026 15:13:44.912143 1655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:44.912458 1655 log.cc:826] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:44.913340 1655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93: No bootstrap required, opened a new log
I20251026 15:13:44.915599 1655 raft_consensus.cc:359] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220b5b30188d42b6983502b5bd18bd93" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45351 } }
I20251026 15:13:44.915779 1655 raft_consensus.cc:385] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:44.915817 1655 raft_consensus.cc:740] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220b5b30188d42b6983502b5bd18bd93, State: Initialized, Role: FOLLOWER
I20251026 15:13:44.915961 1655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220b5b30188d42b6983502b5bd18bd93" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45351 } }
I20251026 15:13:44.916050 1655 raft_consensus.cc:399] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:13:44.916093 1655 raft_consensus.cc:493] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:13:44.916147 1655 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:44.916765 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 1588
I20251026 15:13:44.916868 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:13:44.917028 1655 raft_consensus.cc:515] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220b5b30188d42b6983502b5bd18bd93" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45351 } }
I20251026 15:13:44.917105 15185 external_mini_cluster.cc:1468] Setting key 239636f754821e27b108509d4f14397e
I20251026 15:13:44.917161 1655 leader_election.cc:304] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 220b5b30188d42b6983502b5bd18bd93; no voters:
I20251026 15:13:44.917337 1655 leader_election.cc:290] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:13:44.917385 1658 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:44.917523 1658 raft_consensus.cc:697] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [term 1 LEADER]: Becoming Leader. State: Replica: 220b5b30188d42b6983502b5bd18bd93, State: Running, Role: LEADER
I20251026 15:13:44.917577 1655 sys_catalog.cc:565] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:13:44.917610 1658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220b5b30188d42b6983502b5bd18bd93" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45351 } }
I20251026 15:13:44.917949 1658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "220b5b30188d42b6983502b5bd18bd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220b5b30188d42b6983502b5bd18bd93" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45351 } } }
I20251026 15:13:44.918026 1658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:44.917977 1659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 220b5b30188d42b6983502b5bd18bd93. Latest consensus state: current_term: 1 leader_uuid: "220b5b30188d42b6983502b5bd18bd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220b5b30188d42b6983502b5bd18bd93" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 45351 } } }
I20251026 15:13:44.918193 1659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:44.919363 1665 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:13:44.919842 1665 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:13:44.921300 1665 catalog_manager.cc:1357] Generated new cluster ID: c2b8161c434a49a5b346dbe9c6c81019
I20251026 15:13:44.921346 1665 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:13:44.923579 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:44.917541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.14.212.126:45351)
1026 15:13:44.917914 (+ 373us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:44.917922 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:44.918313 (+ 391us) server_negotiation.cc:408] Connection header received
1026 15:13:44.918783 (+ 470us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:44.918793 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:44.918861 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:44.918998 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:44.919709 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.920876 (+ 1167us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:44.921716 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:44.921951 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:44.922828 (+ 877us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:44.922852 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:44.922853 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:44.923104 (+ 251us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:44.923135 (+ 31us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:44.923143 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:44.923206 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:44.923379 (+ 173us) server_negotiation.cc:300] Negotiation successful
1026 15:13:44.923428 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":92,"threads_started":1}
I20251026 15:13:44.945860 1665 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:13:44.946689 1665 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:13:44.954499 1665 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93: Generated new TSK 0
I20251026 15:13:44.954727 1665 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:13:44.956915 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:45351
--builtin_ntp_servers=127.14.212.84:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251026 15:13:45.042707 1679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:45.042906 1679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:45.042928 1679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:45.044476 1679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:45.044526 1679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:45.044554 1679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:13:45.046116 1679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:45351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:45.046366 1679 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:45.046649 1679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:45.049381 1687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:45.049350 1684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:45.049536 1679 server_base.cc:1047] running on GCE node
W20251026 15:13:45.049350 1685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:45.049882 1679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:45.050141 1679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:45.051302 1679 hybrid_clock.cc:648] HybridClock initialized: now 1761491625051277 us; error 44 us; skew 500 ppm
I20251026 15:13:45.052593 1679 webserver.cc:492] Webserver started at http://127.14.212.65:41725/ using document root <none> and password file <none>
I20251026 15:13:45.052832 1679 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:45.052876 1679 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:45.053078 1679 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:45.054083 1679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "3bb98da997434769812f3e01f6e6d02a"
format_stamp: "Formatted at 2025-10-26 15:13:45 on dist-test-slave-8pgz"
server_key: "69f4426b916705735b5ab1a02e65e5ce"
server_key_iv: "00f008ecdc0f02ef79958b0c414b70b8"
server_key_version: "encryptionkey@0"
I20251026 15:13:45.054425 1679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "3bb98da997434769812f3e01f6e6d02a"
format_stamp: "Formatted at 2025-10-26 15:13:45 on dist-test-slave-8pgz"
server_key: "69f4426b916705735b5ab1a02e65e5ce"
server_key_iv: "00f008ecdc0f02ef79958b0c414b70b8"
server_key_version: "encryptionkey@0"
I20251026 15:13:45.055634 1679 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:45.056330 1693 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:45.056535 1679 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 15:13:45.056629 1679 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "3bb98da997434769812f3e01f6e6d02a"
format_stamp: "Formatted at 2025-10-26 15:13:45 on dist-test-slave-8pgz"
server_key: "69f4426b916705735b5ab1a02e65e5ce"
server_key_iv: "00f008ecdc0f02ef79958b0c414b70b8"
server_key_version: "encryptionkey@0"
I20251026 15:13:45.056699 1679 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:45.072540 1679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:45.072907 1679 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:45.073050 1679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:45.073290 1679 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:45.073630 1679 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:45.073670 1679 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:45.073699 1679 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:45.073724 1679 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:45.081362 1679 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:35407
I20251026 15:13:45.081422 1806 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:35407 every 8 connection(s)
I20251026 15:13:45.081743 1679 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:13:45.086175 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.082509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:32825 (local address 127.14.212.126:45351)
1026 15:13:45.082576 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:45.082579 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:45.082822 (+ 243us) server_negotiation.cc:408] Connection header received
1026 15:13:45.083371 (+ 549us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:45.083372 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:45.083387 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:45.083449 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:45.083927 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.084491 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.085077 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.085253 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.085872 (+ 619us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:45.085877 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:45.085877 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:45.085883 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:45.085898 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:45.085904 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:45.085963 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:45.086051 (+ 88us) server_negotiation.cc:300] Negotiation successful
1026 15:13:45.086090 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251026 15:13:45.086310 1809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.082515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:45351 (local address 127.14.212.65:32825)
1026 15:13:45.082706 (+ 191us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:45.082739 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:45.083271 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:45.083511 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:45.083518 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:45.083527 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:45.083831 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.083833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.084591 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.084592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:45.084993 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.084994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.085131 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.085749 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:13:45.085750 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:45.085771 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:13:45.085790 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:45.085969 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:45.085973 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:45.085976 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:13:45.086053 (+ 77us) client_negotiation.cc:240] Negotiation successful
1026 15:13:45.086133 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20251026 15:13:45.086870 1807 heartbeater.cc:344] Connected to a master server at 127.14.212.126:45351
I20251026 15:13:45.086967 1807 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:45.087181 1807 heartbeater.cc:507] Master 127.14.212.126:45351 requested a full tablet report, sending...
I20251026 15:13:45.087627 1619 ts_manager.cc:194] Registered new tserver with Master: 3bb98da997434769812f3e01f6e6d02a (127.14.212.65:35407)
I20251026 15:13:45.088285 1619 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.14.212.65:32825
I20251026 15:13:45.091768 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 1679
I20251026 15:13:45.091869 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:13:45.092056 15185 external_mini_cluster.cc:1468] Setting key 43de6841bb4d2f5971709b8a044fcfe4
I20251026 15:13:45.093113 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:45351
--builtin_ntp_servers=127.14.212.84:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251026 15:13:45.172669 1810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:45.172875 1810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:45.172899 1810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:45.174355 1810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:45.174427 1810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:45.174455 1810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:13:45.175966 1810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:45351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:45.176200 1810 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:45.176406 1810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:45.179065 1815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:45.179080 1816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:45.179082 1818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:45.179266 1810 server_base.cc:1047] running on GCE node
I20251026 15:13:45.179484 1810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:45.179695 1810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:45.180853 1810 hybrid_clock.cc:648] HybridClock initialized: now 1761491625180832 us; error 35 us; skew 500 ppm
I20251026 15:13:45.182039 1810 webserver.cc:492] Webserver started at http://127.14.212.66:33277/ using document root <none> and password file <none>
I20251026 15:13:45.182264 1810 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:45.182312 1810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:45.182461 1810 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:45.183423 1810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "549038b08771478481d9eb8f7f299cec"
format_stamp: "Formatted at 2025-10-26 15:13:45 on dist-test-slave-8pgz"
server_key: "caaccc64b01ba1f59d00358c74efc9b1"
server_key_iv: "3677b2ad94d69e0edf1014846a6ee189"
server_key_version: "encryptionkey@0"
I20251026 15:13:45.183724 1810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "549038b08771478481d9eb8f7f299cec"
format_stamp: "Formatted at 2025-10-26 15:13:45 on dist-test-slave-8pgz"
server_key: "caaccc64b01ba1f59d00358c74efc9b1"
server_key_iv: "3677b2ad94d69e0edf1014846a6ee189"
server_key_version: "encryptionkey@0"
I20251026 15:13:45.184930 1810 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:13:45.185634 1824 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:45.185824 1810 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:13:45.185896 1810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "549038b08771478481d9eb8f7f299cec"
format_stamp: "Formatted at 2025-10-26 15:13:45 on dist-test-slave-8pgz"
server_key: "caaccc64b01ba1f59d00358c74efc9b1"
server_key_iv: "3677b2ad94d69e0edf1014846a6ee189"
server_key_version: "encryptionkey@0"
I20251026 15:13:45.185961 1810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:45.217350 1810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:45.217682 1810 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:45.217828 1810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:45.218046 1810 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:45.218463 1810 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:45.218503 1810 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:45.218537 1810 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:45.218564 1810 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:45.224318 1810 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:36271
I20251026 15:13:45.224372 1937 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:36271 every 8 connection(s)
I20251026 15:13:45.224687 1810 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:13:45.227662 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 1810
I20251026 15:13:45.227756 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:13:45.227931 15185 external_mini_cluster.cc:1468] Setting key e086e64e9a318bdfb72a1fa65ec5e39b
I20251026 15:13:45.229001 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:45351
--builtin_ntp_servers=127.14.212.84:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251026 15:13:45.229149 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.225417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:56427 (local address 127.14.212.126:45351)
1026 15:13:45.225476 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:45.225478 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:45.225677 (+ 199us) server_negotiation.cc:408] Connection header received
1026 15:13:45.226201 (+ 524us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:45.226202 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:45.226218 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:45.226277 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:45.226820 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.227350 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.228037 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.228193 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.228841 (+ 648us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:45.228845 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:45.228845 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:45.228850 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:45.228866 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:45.228870 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:45.228933 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:45.229014 (+ 81us) server_negotiation.cc:300] Negotiation successful
1026 15:13:45.229051 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251026 15:13:45.229295 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.225396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:45351 (local address 127.14.212.66:56427)
1026 15:13:45.225575 (+ 179us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:45.225600 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:45.226108 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:45.226347 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:45.226350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:45.226360 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:45.226737 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.226741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.227454 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.227455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:45.227948 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.227950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.228102 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.228716 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:13:45.228716 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:45.228744 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:13:45.228763 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:45.228957 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:45.228960 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:45.228962 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:13:45.229016 (+ 54us) client_negotiation.cc:240] Negotiation successful
1026 15:13:45.229094 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20251026 15:13:45.229739 1938 heartbeater.cc:344] Connected to a master server at 127.14.212.126:45351
I20251026 15:13:45.229838 1938 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:45.230082 1938 heartbeater.cc:507] Master 127.14.212.126:45351 requested a full tablet report, sending...
I20251026 15:13:45.230500 1619 ts_manager.cc:194] Registered new tserver with Master: 549038b08771478481d9eb8f7f299cec (127.14.212.66:36271)
I20251026 15:13:45.231060 1619 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.14.212.66:56427
W20251026 15:13:45.311163 1941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:45.311400 1941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:45.311436 1941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:45.313241 1941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:45.313319 1941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:45.313364 1941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:13:45.314893 1941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:45351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:45.315107 1941 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:45.315318 1941 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:45.317929 1947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:45.317924 1946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:45.317958 1949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:45.318138 1941 server_base.cc:1047] running on GCE node
I20251026 15:13:45.318353 1941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:45.318601 1941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:45.319761 1941 hybrid_clock.cc:648] HybridClock initialized: now 1761491625319700 us; error 79 us; skew 500 ppm
I20251026 15:13:45.320976 1941 webserver.cc:492] Webserver started at http://127.14.212.67:39993/ using document root <none> and password file <none>
I20251026 15:13:45.321182 1941 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:45.321231 1941 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:45.321338 1941 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:45.322289 1941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "905b675188b149bda4e18245eec25116"
format_stamp: "Formatted at 2025-10-26 15:13:45 on dist-test-slave-8pgz"
server_key: "1b3e47cf64b5bc052b3850cdf665126a"
server_key_iv: "cad51148976ce360433fa4bced8288ff"
server_key_version: "encryptionkey@0"
I20251026 15:13:45.322635 1941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "905b675188b149bda4e18245eec25116"
format_stamp: "Formatted at 2025-10-26 15:13:45 on dist-test-slave-8pgz"
server_key: "1b3e47cf64b5bc052b3850cdf665126a"
server_key_iv: "cad51148976ce360433fa4bced8288ff"
server_key_version: "encryptionkey@0"
I20251026 15:13:45.323890 1941 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:45.324638 1955 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:45.324819 1941 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:13:45.324894 1941 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "905b675188b149bda4e18245eec25116"
format_stamp: "Formatted at 2025-10-26 15:13:45 on dist-test-slave-8pgz"
server_key: "1b3e47cf64b5bc052b3850cdf665126a"
server_key_iv: "cad51148976ce360433fa4bced8288ff"
server_key_version: "encryptionkey@0"
I20251026 15:13:45.324982 1941 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:45.335840 1941 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:45.336177 1941 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:45.336305 1941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:45.336509 1941 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:45.336822 1941 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:45.336855 1941 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:45.336891 1941 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:45.336916 1941 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:45.342834 1941 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:39991
I20251026 15:13:45.342926 2068 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:39991 every 8 connection(s)
I20251026 15:13:45.343230 1941 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:13:45.343876 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 1941
I20251026 15:13:45.343984 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:13:45.344226 15185 external_mini_cluster.cc:1468] Setting key 31146de54e9f962f01127ae7dc4f3840
I20251026 15:13:45.352105 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.344100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:57203 (local address 127.14.212.126:45351)
1026 15:13:45.344172 (+ 72us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:45.344174 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:45.344189 (+ 15us) server_negotiation.cc:408] Connection header received
1026 15:13:45.344741 (+ 552us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:45.344741 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:45.344752 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:45.344807 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:45.345378 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.345859 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.346605 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.346721 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.351775 (+ 5054us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:45.351780 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:45.351781 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:45.351786 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:45.351804 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:45.351807 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:45.351857 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:45.352006 (+ 149us) server_negotiation.cc:300] Negotiation successful
1026 15:13:45.352044 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251026 15:13:45.352246 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.343869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:45351 (local address 127.14.212.67:57203)
1026 15:13:45.344056 (+ 187us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:45.344089 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:45.344647 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:45.344902 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:45.344906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:45.344917 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:45.345302 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.345306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.346046 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.346047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:45.346524 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.346527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.346644 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.351615 (+ 4971us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:13:45.351615 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:45.351643 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:13:45.351666 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:45.351915 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:45.351921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:45.351924 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:13:45.352006 (+ 82us) client_negotiation.cc:240] Negotiation successful
1026 15:13:45.352085 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20251026 15:13:45.352751 2069 heartbeater.cc:344] Connected to a master server at 127.14.212.126:45351
I20251026 15:13:45.352843 2069 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:45.353077 2069 heartbeater.cc:507] Master 127.14.212.126:45351 requested a full tablet report, sending...
I20251026 15:13:45.353462 1619 ts_manager.cc:194] Registered new tserver with Master: 905b675188b149bda4e18245eec25116 (127.14.212.67:39991)
I20251026 15:13:45.353896 1619 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.14.212.67:57203
I20251026 15:13:45.355973 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:13:45.360217 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.357380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.14.212.126:45351)
1026 15:13:45.357472 (+ 92us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:45.357475 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:45.357546 (+ 71us) server_negotiation.cc:408] Connection header received
1026 15:13:45.357624 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:45.357625 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:45.357635 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:45.357690 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:45.358005 (+ 315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.358533 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.359123 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.359337 (+ 214us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.359769 (+ 432us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:45.359773 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:45.359773 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:45.359778 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:45.359792 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:45.359796 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:45.359890 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:45.360017 (+ 127us) server_negotiation.cc:300] Negotiation successful
1026 15:13:45.360031 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251026 15:13:45.361828 1619 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:13:45.363226 1619 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:13:45.368832 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.365377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:36271 (local address 127.0.0.1:41686)
1026 15:13:45.365710 (+ 333us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:45.365720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:45.365781 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:45.366165 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:45.366166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:45.366167 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:45.366424 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.366425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.367553 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.367554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:45.368487 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.368490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.368563 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.368607 (+ 44us) client_negotiation.cc:769] Sending connection context
1026 15:13:45.368660 (+ 53us) client_negotiation.cc:240] Negotiation successful
1026 15:13:45.368697 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":62,"threads_started":1}
I20251026 15:13:45.368836 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.365156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:35407 (local address 127.0.0.1:39214)
1026 15:13:45.365354 (+ 198us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:45.365383 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:45.365463 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:45.366162 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:45.366163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:45.366164 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:45.366385 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.366386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.367892 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.367893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:45.368604 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.368607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.368674 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.368677 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:13:45.368709 (+ 32us) client_negotiation.cc:240] Negotiation successful
1026 15:13:45.368746 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
I20251026 15:13:45.369331 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.365566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.14.212.66:36271)
1026 15:13:45.365723 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:45.365735 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:45.365767 (+ 32us) server_negotiation.cc:408] Connection header received
1026 15:13:45.365859 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:45.365861 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:45.365877 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:45.366080 (+ 203us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:45.366506 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.367455 (+ 949us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.368584 (+ 1129us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.369106 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.369199 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:45.369228 (+ 29us) server_negotiation.cc:300] Negotiation successful
1026 15:13:45.369263 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20251026 15:13:45.369499 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.365384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.14.212.67:39991)
1026 15:13:45.365546 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:45.365558 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:45.365568 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:13:45.365603 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:45.365605 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:45.365618 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:45.365682 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:45.365964 (+ 282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.367562 (+ 1598us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.368738 (+ 1176us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.369257 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.369347 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:45.369381 (+ 34us) server_negotiation.cc:300] Negotiation successful
1026 15:13:45.369421 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20251026 15:13:45.369530 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.365275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.14.212.65:35407)
1026 15:13:45.365916 (+ 641us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:45.365923 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:45.365933 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:13:45.365983 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:45.365985 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:45.366002 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:45.366067 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:45.366480 (+ 413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.367817 (+ 1337us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.368868 (+ 1051us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.369342 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.369416 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:45.369448 (+ 32us) server_negotiation.cc:300] Negotiation successful
1026 15:13:45.369479 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":80,"threads_started":1}
I20251026 15:13:45.369903 1872 tablet_service.cc:1505] Processing CreateTablet for tablet ca7c4f61da034d078cefb105a731ece8 (DEFAULT_TABLE table=test-table [id=123bed599b9245a68f7999b9602b4be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:45.370160 1872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca7c4f61da034d078cefb105a731ece8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:45.369688 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.365203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:39991 (local address 127.0.0.1:35278)
1026 15:13:45.365432 (+ 229us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:45.365443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:45.365495 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:45.365698 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:45.365700 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:45.365702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:45.365884 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.365886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.367652 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.367653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:45.368562 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.368563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.368617 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.368619 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:13:45.368657 (+ 38us) client_negotiation.cc:240] Negotiation successful
1026 15:13:45.368698 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":97,"threads_started":1}
I20251026 15:13:45.371229 2003 tablet_service.cc:1505] Processing CreateTablet for tablet ca7c4f61da034d078cefb105a731ece8 (DEFAULT_TABLE table=test-table [id=123bed599b9245a68f7999b9602b4be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:45.371480 2003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca7c4f61da034d078cefb105a731ece8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:45.372565 2083 tablet_bootstrap.cc:492] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec: Bootstrap starting.
I20251026 15:13:45.373044 1741 tablet_service.cc:1505] Processing CreateTablet for tablet ca7c4f61da034d078cefb105a731ece8 (DEFAULT_TABLE table=test-table [id=123bed599b9245a68f7999b9602b4be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:45.373253 2083 tablet_bootstrap.cc:654] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:45.373301 1741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca7c4f61da034d078cefb105a731ece8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:45.373548 2083 log.cc:826] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:45.374001 2085 tablet_bootstrap.cc:492] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116: Bootstrap starting.
I20251026 15:13:45.374253 2083 tablet_bootstrap.cc:492] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec: No bootstrap required, opened a new log
I20251026 15:13:45.374316 2083 ts_tablet_manager.cc:1403] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:13:45.374697 2085 tablet_bootstrap.cc:654] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:45.374993 2085 log.cc:826] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:45.375661 2083 raft_consensus.cc:359] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb98da997434769812f3e01f6e6d02a" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 35407 } } peers { permanent_uuid: "549038b08771478481d9eb8f7f299cec" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36271 } } peers { permanent_uuid: "905b675188b149bda4e18245eec25116" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 39991 } }
I20251026 15:13:45.375752 2085 tablet_bootstrap.cc:492] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116: No bootstrap required, opened a new log
I20251026 15:13:45.375777 2083 raft_consensus.cc:385] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:45.375804 2085 ts_tablet_manager.cc:1403] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 15:13:45.375809 2083 raft_consensus.cc:740] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549038b08771478481d9eb8f7f299cec, State: Initialized, Role: FOLLOWER
I20251026 15:13:45.375905 2083 consensus_queue.cc:260] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb98da997434769812f3e01f6e6d02a" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 35407 } } peers { permanent_uuid: "549038b08771478481d9eb8f7f299cec" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36271 } } peers { permanent_uuid: "905b675188b149bda4e18245eec25116" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 39991 } }
I20251026 15:13:45.376236 2083 ts_tablet_manager.cc:1434] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:13:45.376284 2087 tablet_bootstrap.cc:492] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a: Bootstrap starting.
I20251026 15:13:45.376416 1938 heartbeater.cc:499] Master 127.14.212.126:45351 was elected leader, sending a full tablet report...
I20251026 15:13:45.377000 2087 tablet_bootstrap.cc:654] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:45.377360 2087 log.cc:826] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:45.377388 2085 raft_consensus.cc:359] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb98da997434769812f3e01f6e6d02a" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 35407 } } peers { permanent_uuid: "549038b08771478481d9eb8f7f299cec" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36271 } } peers { permanent_uuid: "905b675188b149bda4e18245eec25116" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 39991 } }
I20251026 15:13:45.377532 2085 raft_consensus.cc:385] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:45.377575 2085 raft_consensus.cc:740] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905b675188b149bda4e18245eec25116, State: Initialized, Role: FOLLOWER
I20251026 15:13:45.377666 2085 consensus_queue.cc:260] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb98da997434769812f3e01f6e6d02a" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 35407 } } peers { permanent_uuid: "549038b08771478481d9eb8f7f299cec" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36271 } } peers { permanent_uuid: "905b675188b149bda4e18245eec25116" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 39991 } }
I20251026 15:13:45.377954 2085 ts_tablet_manager.cc:1434] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:13:45.377983 2069 heartbeater.cc:499] Master 127.14.212.126:45351 was elected leader, sending a full tablet report...
I20251026 15:13:45.378114 2087 tablet_bootstrap.cc:492] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a: No bootstrap required, opened a new log
I20251026 15:13:45.378182 2087 ts_tablet_manager.cc:1403] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:13:45.379411 2087 raft_consensus.cc:359] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb98da997434769812f3e01f6e6d02a" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 35407 } } peers { permanent_uuid: "549038b08771478481d9eb8f7f299cec" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36271 } } peers { permanent_uuid: "905b675188b149bda4e18245eec25116" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 39991 } }
I20251026 15:13:45.379526 2087 raft_consensus.cc:385] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:45.379545 2087 raft_consensus.cc:740] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb98da997434769812f3e01f6e6d02a, State: Initialized, Role: FOLLOWER
I20251026 15:13:45.379616 2087 consensus_queue.cc:260] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb98da997434769812f3e01f6e6d02a" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 35407 } } peers { permanent_uuid: "549038b08771478481d9eb8f7f299cec" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36271 } } peers { permanent_uuid: "905b675188b149bda4e18245eec25116" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 39991 } }
I20251026 15:13:45.379823 2087 ts_tablet_manager.cc:1434] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:13:45.379882 1807 heartbeater.cc:499] Master 127.14.212.126:45351 was elected leader, sending a full tablet report...
W20251026 15:13:45.475232 1939 tablet.cc:2378] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 15:13:45.582517 1808 tablet.cc:2378] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251026 15:13:45.593708 2070 tablet.cc:2378] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:13:45.645401 2090 raft_consensus.cc:493] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:13:45.645535 2090 raft_consensus.cc:515] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb98da997434769812f3e01f6e6d02a" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 35407 } } peers { permanent_uuid: "549038b08771478481d9eb8f7f299cec" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36271 } } peers { permanent_uuid: "905b675188b149bda4e18245eec25116" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 39991 } }
I20251026 15:13:45.645928 2090 leader_election.cc:290] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bb98da997434769812f3e01f6e6d02a (127.14.212.65:35407), 549038b08771478481d9eb8f7f299cec (127.14.212.66:36271)
I20251026 15:13:45.648221 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.646164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:35407 (local address 127.14.212.67:59309)
1026 15:13:45.646250 (+ 86us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:45.646273 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:45.646348 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:45.646571 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:45.646572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:45.646574 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:45.646745 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.646747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.647432 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.647433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:45.647994 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.647996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.648066 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.648070 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:45.648103 (+ 33us) client_negotiation.cc:240] Negotiation successful
1026 15:13:45.648148 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20251026 15:13:45.648700 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.646280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:59309 (local address 127.14.212.65:35407)
1026 15:13:45.646339 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:45.646341 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:45.646353 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:13:45.646486 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:45.646487 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:45.646499 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:45.646550 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:45.646821 (+ 271us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.647340 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.648079 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.648577 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.648587 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:45.648610 (+ 23us) server_negotiation.cc:300] Negotiation successful
1026 15:13:45.648638 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251026 15:13:45.648980 1761 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca7c4f61da034d078cefb105a731ece8" candidate_uuid: "905b675188b149bda4e18245eec25116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb98da997434769812f3e01f6e6d02a" is_pre_election: true
I20251026 15:13:45.649151 1761 raft_consensus.cc:2468] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905b675188b149bda4e18245eec25116 in term 0.
I20251026 15:13:45.649299 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.646382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:55659 (local address 127.14.212.66:36271)
1026 15:13:45.646453 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:45.646455 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:45.646574 (+ 119us) server_negotiation.cc:408] Connection header received
1026 15:13:45.646613 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:45.646614 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:45.646624 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:45.646680 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:45.647213 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.647716 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.648601 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.649056 (+ 455us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.649064 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:45.649186 (+ 122us) server_negotiation.cc:300] Negotiation successful
1026 15:13:45.649233 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251026 15:13:45.649370 1956 leader_election.cc:304] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bb98da997434769812f3e01f6e6d02a, 905b675188b149bda4e18245eec25116; no voters:
I20251026 15:13:45.649518 2090 raft_consensus.cc:2804] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:13:45.649540 2092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.646311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:36271 (local address 127.14.212.67:55659)
1026 15:13:45.646479 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:45.646493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:45.646561 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:45.646693 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:45.646693 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:45.646695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:45.647117 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.647119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.647825 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.647826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:45.648470 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:45.648472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.649096 (+ 624us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.649100 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:45.649390 (+ 290us) client_negotiation.cc:240] Negotiation successful
1026 15:13:45.649443 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":123,"threads_started":1}
I20251026 15:13:45.649566 2090 raft_consensus.cc:493] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:13:45.649595 2090 raft_consensus.cc:3060] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:45.649850 1892 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca7c4f61da034d078cefb105a731ece8" candidate_uuid: "905b675188b149bda4e18245eec25116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549038b08771478481d9eb8f7f299cec" is_pre_election: true
I20251026 15:13:45.649964 1892 raft_consensus.cc:2468] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905b675188b149bda4e18245eec25116 in term 0.
I20251026 15:13:45.650430 2090 raft_consensus.cc:515] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb98da997434769812f3e01f6e6d02a" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 35407 } } peers { permanent_uuid: "549038b08771478481d9eb8f7f299cec" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36271 } } peers { permanent_uuid: "905b675188b149bda4e18245eec25116" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 39991 } }
I20251026 15:13:45.650555 2090 leader_election.cc:290] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [CANDIDATE]: Term 1 election: Requested vote from peers 3bb98da997434769812f3e01f6e6d02a (127.14.212.65:35407), 549038b08771478481d9eb8f7f299cec (127.14.212.66:36271)
I20251026 15:13:45.650732 1761 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca7c4f61da034d078cefb105a731ece8" candidate_uuid: "905b675188b149bda4e18245eec25116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb98da997434769812f3e01f6e6d02a"
I20251026 15:13:45.650759 1892 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca7c4f61da034d078cefb105a731ece8" candidate_uuid: "905b675188b149bda4e18245eec25116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549038b08771478481d9eb8f7f299cec"
I20251026 15:13:45.650810 1761 raft_consensus.cc:3060] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:45.650828 1892 raft_consensus.cc:3060] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:45.651536 1761 raft_consensus.cc:2468] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905b675188b149bda4e18245eec25116 in term 1.
I20251026 15:13:45.651722 1956 leader_election.cc:304] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bb98da997434769812f3e01f6e6d02a, 905b675188b149bda4e18245eec25116; no voters:
I20251026 15:13:45.651813 2090 raft_consensus.cc:2804] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:45.651968 2090 raft_consensus.cc:697] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [term 1 LEADER]: Becoming Leader. State: Replica: 905b675188b149bda4e18245eec25116, State: Running, Role: LEADER
I20251026 15:13:45.652073 2090 consensus_queue.cc:237] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb98da997434769812f3e01f6e6d02a" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 35407 } } peers { permanent_uuid: "549038b08771478481d9eb8f7f299cec" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36271 } } peers { permanent_uuid: "905b675188b149bda4e18245eec25116" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 39991 } }
I20251026 15:13:45.652232 1892 raft_consensus.cc:2468] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905b675188b149bda4e18245eec25116 in term 1.
I20251026 15:13:45.652849 1619 catalog_manager.cc:5649] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 reported cstate change: term changed from 0 to 1, leader changed from <none> to 905b675188b149bda4e18245eec25116 (127.14.212.67). New cstate: current_term: 1 leader_uuid: "905b675188b149bda4e18245eec25116" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb98da997434769812f3e01f6e6d02a" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 35407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "549038b08771478481d9eb8f7f299cec" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "905b675188b149bda4e18245eec25116" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 39991 } health_report { overall_health: HEALTHY } } }
I20251026 15:13:45.723528 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.721317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.14.212.67:39991)
1026 15:13:45.721376 (+ 59us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:45.721378 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:45.721386 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:13:45.721426 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:45.721427 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:45.721443 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:45.721497 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:13:45.721857 (+ 360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.722373 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:45.722974 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:45.723106 (+ 132us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:45.723255 (+ 149us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:13:45.723337 (+ 82us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:13:45.723398 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:45.723461 (+ 63us) server_negotiation.cc:300] Negotiation successful
1026 15:13:45.723471 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251026 15:13:45.725203 1761 raft_consensus.cc:1275] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a [term 1 FOLLOWER]: Refusing update from remote peer 905b675188b149bda4e18245eec25116: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:45.725224 1892 raft_consensus.cc:1275] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec [term 1 FOLLOWER]: Refusing update from remote peer 905b675188b149bda4e18245eec25116: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:45.725541 2093 consensus_queue.cc:1048] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [LEADER]: Connected to new peer: Peer: permanent_uuid: "549038b08771478481d9eb8f7f299cec" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:45.725663 2090 consensus_queue.cc:1048] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb98da997434769812f3e01f6e6d02a" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:45.728200 2096 mvcc.cc:204] Tried to move back new op lower bound from 7215069698967572480 to 7215069698671345664. Current Snapshot: MvccSnapshot[applied={T|T < 7215069698967572480}]
I20251026 15:13:45.730655 1618 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47340:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:45.730744 1618 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47340:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:45.731431 1618 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 220b5b30188d42b6983502b5bd18bd93: Sending DeleteTablet for 3 replicas of tablet ca7c4f61da034d078cefb105a731ece8
I20251026 15:13:45.731760 1741 tablet_service.cc:1552] Processing DeleteTablet for tablet ca7c4f61da034d078cefb105a731ece8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:45 UTC) from {username='slave'} at 127.0.0.1:39214
I20251026 15:13:45.731767 2003 tablet_service.cc:1552] Processing DeleteTablet for tablet ca7c4f61da034d078cefb105a731ece8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:45 UTC) from {username='slave'} at 127.0.0.1:35278
I20251026 15:13:45.731793 1872 tablet_service.cc:1552] Processing DeleteTablet for tablet ca7c4f61da034d078cefb105a731ece8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:45 UTC) from {username='slave'} at 127.0.0.1:41686
I20251026 15:13:45.732036 2107 tablet_replica.cc:333] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a: stopping tablet replica
I20251026 15:13:45.732152 2106 tablet_replica.cc:333] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec: stopping tablet replica
I20251026 15:13:45.732168 2107 raft_consensus.cc:2243] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:45.732230 2106 raft_consensus.cc:2243] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:45.732272 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 1679
I20251026 15:13:45.732331 2106 raft_consensus.cc:2272] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:45.732345 2107 raft_consensus.cc:2272] T ca7c4f61da034d078cefb105a731ece8 P 3bb98da997434769812f3e01f6e6d02a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:45.732501 2105 tablet_replica.cc:333] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116: stopping tablet replica
I20251026 15:13:45.732568 2105 raft_consensus.cc:2243] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:13:45.732642 2105 raft_consensus.cc:2272] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:45.732851 2106 ts_tablet_manager.cc:1916] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:45.734469 2105 ts_tablet_manager.cc:1916] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:45.734949 2106 ts_tablet_manager.cc:1929] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:45.735023 2106 log.cc:1199] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/ca7c4f61da034d078cefb105a731ece8
I20251026 15:13:45.735282 2106 ts_tablet_manager.cc:1950] T ca7c4f61da034d078cefb105a731ece8 P 549038b08771478481d9eb8f7f299cec: Deleting consensus metadata
I20251026 15:13:45.735770 1603 catalog_manager.cc:4985] TS 549038b08771478481d9eb8f7f299cec (127.14.212.66:36271): tablet ca7c4f61da034d078cefb105a731ece8 (table test-table [id=123bed599b9245a68f7999b9602b4be8]) successfully deleted
I20251026 15:13:45.736122 2105 ts_tablet_manager.cc:1929] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:45.736195 2105 log.cc:1199] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/ca7c4f61da034d078cefb105a731ece8
I20251026 15:13:45.736465 2105 ts_tablet_manager.cc:1950] T ca7c4f61da034d078cefb105a731ece8 P 905b675188b149bda4e18245eec25116: Deleting consensus metadata
I20251026 15:13:45.736883 1604 catalog_manager.cc:4985] TS 905b675188b149bda4e18245eec25116 (127.14.212.67:39991): tablet ca7c4f61da034d078cefb105a731ece8 (table test-table [id=123bed599b9245a68f7999b9602b4be8]) successfully deleted
W20251026 15:13:45.740841 1603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:35407 (error 108)
I20251026 15:13:45.741103 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 1810
I20251026 15:13:45.741660 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:45.741261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:35407 (local address 127.0.0.1:39226)
1026 15:13:45.741584 (+ 323us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:45.741622 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:35407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
W20251026 15:13:45.741997 1603 catalog_manager.cc:4712] TS 3bb98da997434769812f3e01f6e6d02a (127.14.212.65:35407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca7c4f61da034d078cefb105a731ece8: Network error: Client connection negotiation failed: client connection to 127.14.212.65:35407: connect: Connection refused (error 111)
I20251026 15:13:45.745957 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 1941
I20251026 15:13:45.750756 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 1588
2025-10-26T15:13:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (994 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:13:45 dist-test-slave-8pgz krb5kdc[2111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:13:45 dist-test-slave-8pgz krb5kdc[2111](info): set up 2 sockets
Oct 26 15:13:45 dist-test-slave-8pgz krb5kdc[2111](info): commencing operation
krb5kdc: starting...
W20251026 15:13:47.747753 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.977s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:13:47 dist-test-slave-8pgz krb5kdc[2111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:13:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:13:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:47.885066 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:43475
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:43475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:47.968760 2127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:47.968958 2127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:47.968983 2127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:47.970530 2127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:13:47.970582 2127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:47.970597 2127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:47.970609 2127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:13:47.970623 2127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:13:47.972383 2127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:43475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:43475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:47.972610 2127 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:47.972815 2127 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:47.975476 2132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:47.975492 2135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:47.975477 2133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:47.975711 2127 server_base.cc:1047] running on GCE node
I20251026 15:13:47.976009 2127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:47.976238 2127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:47.977396 2127 hybrid_clock.cc:648] HybridClock initialized: now 1761491627977370 us; error 38 us; skew 500 ppm
Oct 26 15:13:47 dist-test-slave-8pgz krb5kdc[2111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491627, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:47.979961 2127 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:13:47.980487 2127 webserver.cc:492] Webserver started at http://127.14.212.126:34369/ using document root <none> and password file <none>
I20251026 15:13:47.980654 2127 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:47.980695 2127 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:47.980779 2127 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:47.981762 2127 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "8acd68391a784cb5943fa6d1aaa5afc9"
format_stamp: "Formatted at 2025-10-26 15:13:47 on dist-test-slave-8pgz"
server_key: "cf09d0d97913b22f2acbe4ea9b1221a4"
server_key_iv: "58ce181763d5189005aa4c2cf1b83f7c"
server_key_version: "encryptionkey@0"
I20251026 15:13:47.982056 2127 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "8acd68391a784cb5943fa6d1aaa5afc9"
format_stamp: "Formatted at 2025-10-26 15:13:47 on dist-test-slave-8pgz"
server_key: "cf09d0d97913b22f2acbe4ea9b1221a4"
server_key_iv: "58ce181763d5189005aa4c2cf1b83f7c"
server_key_version: "encryptionkey@0"
I20251026 15:13:47.983390 2127 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:13:47.984134 2142 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:47.984351 2127 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:47.984431 2127 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "8acd68391a784cb5943fa6d1aaa5afc9"
format_stamp: "Formatted at 2025-10-26 15:13:47 on dist-test-slave-8pgz"
server_key: "cf09d0d97913b22f2acbe4ea9b1221a4"
server_key_iv: "58ce181763d5189005aa4c2cf1b83f7c"
server_key_version: "encryptionkey@0"
I20251026 15:13:47.984499 2127 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:48.003260 2127 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:48.005338 2127 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:48.005470 2127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:48.009449 2127 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:43475
I20251026 15:13:48.009483 2194 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:43475 every 8 connection(s)
I20251026 15:13:48.009817 2127 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:13:48.010429 2195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:48.012823 2195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9: Bootstrap starting.
I20251026 15:13:48.013391 2195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:48.013600 2195 log.cc:826] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:48.014350 2195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9: No bootstrap required, opened a new log
I20251026 15:13:48.015606 2195 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8acd68391a784cb5943fa6d1aaa5afc9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43475 } }
I20251026 15:13:48.015738 2195 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:48.015758 2195 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8acd68391a784cb5943fa6d1aaa5afc9, State: Initialized, Role: FOLLOWER
I20251026 15:13:48.015851 2195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8acd68391a784cb5943fa6d1aaa5afc9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43475 } }
I20251026 15:13:48.015940 2195 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:13:48.015971 2195 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:13:48.016002 2195 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:48.016530 2195 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8acd68391a784cb5943fa6d1aaa5afc9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43475 } }
I20251026 15:13:48.016620 2195 leader_election.cc:304] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8acd68391a784cb5943fa6d1aaa5afc9; no voters:
I20251026 15:13:48.016741 2195 leader_election.cc:290] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:13:48.016814 2198 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:48.016969 2195 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:13:48.016981 2198 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [term 1 LEADER]: Becoming Leader. State: Replica: 8acd68391a784cb5943fa6d1aaa5afc9, State: Running, Role: LEADER
I20251026 15:13:48.017096 2198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8acd68391a784cb5943fa6d1aaa5afc9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43475 } }
I20251026 15:13:48.017467 2199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8acd68391a784cb5943fa6d1aaa5afc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8acd68391a784cb5943fa6d1aaa5afc9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43475 } } }
I20251026 15:13:48.017504 2200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8acd68391a784cb5943fa6d1aaa5afc9. Latest consensus state: current_term: 1 leader_uuid: "8acd68391a784cb5943fa6d1aaa5afc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8acd68391a784cb5943fa6d1aaa5afc9" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43475 } } }
I20251026 15:13:48.017555 2199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:48.017570 2200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:48.017809 2205 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:13:48.018304 2205 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:13:48.019809 2205 catalog_manager.cc:1357] Generated new cluster ID: b1050757870347cba79be982c6d97d29
I20251026 15:13:48.019870 2205 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:13:48.020035 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 2127
I20251026 15:13:48.020154 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:13:48.020328 15185 external_mini_cluster.cc:1468] Setting key e523faf35339980500e1cec0b1380b8e
Oct 26 15:13:48 dist-test-slave-8pgz krb5kdc[2111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:48.032824 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.020886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.14.212.126:43475)
1026 15:13:48.022297 (+ 1411us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.022308 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.022332 (+ 24us) server_negotiation.cc:408] Connection header received
1026 15:13:48.022468 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.022474 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.022526 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.022649 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:48.023603 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.024791 (+ 1188us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.026377 (+ 1586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.026671 (+ 294us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.028915 (+ 2244us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:48.028922 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:48.028922 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:48.028949 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:48.031143 (+ 2194us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:48.031500 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:48.031503 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:48.031503 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:48.031564 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:48.031695 (+ 131us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:48.031712 (+ 17us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:48.031712 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:48.031971 (+ 259us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:48.032097 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.032513 (+ 416us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.032639 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1368,"thread_start_us":81,"threads_started":1}
I20251026 15:13:48.035405 2205 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:13:48.035926 2205 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:13:48.040421 2205 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9: Generated new TSK 0
I20251026 15:13:48.040591 2205 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:48.091228 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:43475
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:48.172525 2223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:48.172730 2223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:48.172770 2223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:48.174288 2223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:48.174337 2223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:48.174381 2223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:13:48.176074 2223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:48.176337 2223 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:48.176522 2223 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:48.179167 2231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:48.179165 2228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:48.179168 2229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:48.179394 2223 server_base.cc:1047] running on GCE node
I20251026 15:13:48.179598 2223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:48.179811 2223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:48.180946 2223 hybrid_clock.cc:648] HybridClock initialized: now 1761491628180917 us; error 44 us; skew 500 ppm
Oct 26 15:13:48 dist-test-slave-8pgz krb5kdc[2111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491628, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:48.183467 2223 init.cc:377] Logged in from keytab as kudu/127.14.212.65@KRBTEST.COM (short username kudu)
I20251026 15:13:48.183960 2223 webserver.cc:492] Webserver started at http://127.14.212.65:35133/ using document root <none> and password file <none>
I20251026 15:13:48.184124 2223 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:48.184155 2223 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:48.184242 2223 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:48.185148 2223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "6c2c4913e51b4baa985f7ae9126d42ce"
format_stamp: "Formatted at 2025-10-26 15:13:48 on dist-test-slave-8pgz"
server_key: "f644e2b289b9566f9f4eccc411c2f5ad"
server_key_iv: "19018a532656dc9adfcab07a52269876"
server_key_version: "encryptionkey@0"
I20251026 15:13:48.185425 2223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "6c2c4913e51b4baa985f7ae9126d42ce"
format_stamp: "Formatted at 2025-10-26 15:13:48 on dist-test-slave-8pgz"
server_key: "f644e2b289b9566f9f4eccc411c2f5ad"
server_key_iv: "19018a532656dc9adfcab07a52269876"
server_key_version: "encryptionkey@0"
I20251026 15:13:48.186702 2223 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:13:48.187384 2238 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:48.187593 2223 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251026 15:13:48.187674 2223 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "6c2c4913e51b4baa985f7ae9126d42ce"
format_stamp: "Formatted at 2025-10-26 15:13:48 on dist-test-slave-8pgz"
server_key: "f644e2b289b9566f9f4eccc411c2f5ad"
server_key_iv: "19018a532656dc9adfcab07a52269876"
server_key_version: "encryptionkey@0"
I20251026 15:13:48.187732 2223 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:48.200793 2223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:48.202805 2223 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:48.202953 2223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:48.203157 2223 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:48.203444 2223 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:48.203474 2223 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:48.203497 2223 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:48.203516 2223 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:48.208842 2223 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:43679
I20251026 15:13:48.208969 2351 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:43679 every 8 connection(s)
I20251026 15:13:48.209203 2223 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
Oct 26 15:13:48 dist-test-slave-8pgz krb5kdc[2111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491628, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.65@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:48.216110 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 2223
I20251026 15:13:48.216226 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:13:48.216420 15185 external_mini_cluster.cc:1468] Setting key dc6ec898a3937c45b564e6ee3be8df87
I20251026 15:13:48.217710 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.209842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43475 (local address 127.14.212.65:57745)
1026 15:13:48.209963 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:48.209988 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:48.210505 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:48.210695 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:48.210698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:48.211003 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:48.211304 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.211307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.212095 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.212096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:48.212496 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.212497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.212614 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.213482 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:48.213497 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:48.214912 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:48.216425 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:48.216432 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:48.216436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:48.216684 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:48.216831 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:48.216832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:48.216832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:48.216905 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:48.217133 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:48.217138 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:48.217304 (+ 166us) client_negotiation.cc:769] Sending connection context
1026 15:13:48.217430 (+ 126us) client_negotiation.cc:240] Negotiation successful
1026 15:13:48.217549 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
I20251026 15:13:48.217748 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.209894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:57745 (local address 127.14.212.126:43475)
1026 15:13:48.209978 (+ 84us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.209980 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.210068 (+ 88us) server_negotiation.cc:408] Connection header received
1026 15:13:48.210579 (+ 511us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.210581 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.210599 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.210653 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:48.211418 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.211926 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.212562 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.212668 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.215017 (+ 2349us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:48.215021 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:48.215022 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:48.215029 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:48.216331 (+ 1302us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:48.216740 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:48.216741 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:48.216741 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:48.216769 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:48.216961 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:48.216961 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:48.216962 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:48.217064 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:48.217133 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.217476 (+ 343us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.217586 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251026 15:13:48.218571 2352 heartbeater.cc:344] Connected to a master server at 127.14.212.126:43475
I20251026 15:13:48.218703 2352 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:48.218951 2352 heartbeater.cc:507] Master 127.14.212.126:43475 requested a full tablet report, sending...
I20251026 15:13:48.219440 2159 ts_manager.cc:194] Registered new tserver with Master: 6c2c4913e51b4baa985f7ae9126d42ce (127.14.212.65:43679)
I20251026 15:13:48.220041 2159 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.65@KRBTEST.COM'} at 127.14.212.65:57745
WARNING: no policy specified for kudu/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:48.262552 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:43475
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:48.341570 2359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:48.341763 2359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:48.341789 2359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:48.343176 2359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:48.343223 2359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:48.343250 2359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:13:48.344834 2359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:48.345017 2359 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:48.345170 2359 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:48.347774 2364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:48.347954 2367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:48.347949 2365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:48.347997 2359 server_base.cc:1047] running on GCE node
I20251026 15:13:48.348274 2359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:48.348490 2359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:48.349622 2359 hybrid_clock.cc:648] HybridClock initialized: now 1761491628349593 us; error 49 us; skew 500 ppm
Oct 26 15:13:48 dist-test-slave-8pgz krb5kdc[2111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491628, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:48.352236 2359 init.cc:377] Logged in from keytab as kudu/127.14.212.66@KRBTEST.COM (short username kudu)
I20251026 15:13:48.352746 2359 webserver.cc:492] Webserver started at http://127.14.212.66:41615/ using document root <none> and password file <none>
I20251026 15:13:48.352919 2359 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:48.352952 2359 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:48.353034 2359 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:48.353948 2359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "d9a56999ddca4395ad0c4160da91bb93"
format_stamp: "Formatted at 2025-10-26 15:13:48 on dist-test-slave-8pgz"
server_key: "c8d02ad0fc61c70498df51c2ce766c06"
server_key_iv: "ee0f7ee56a4650f575e9fd26ba676a17"
server_key_version: "encryptionkey@0"
I20251026 15:13:48.354209 2359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "d9a56999ddca4395ad0c4160da91bb93"
format_stamp: "Formatted at 2025-10-26 15:13:48 on dist-test-slave-8pgz"
server_key: "c8d02ad0fc61c70498df51c2ce766c06"
server_key_iv: "ee0f7ee56a4650f575e9fd26ba676a17"
server_key_version: "encryptionkey@0"
I20251026 15:13:48.355401 2359 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:13:48.356163 2374 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:48.356375 2359 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:48.356451 2359 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "d9a56999ddca4395ad0c4160da91bb93"
format_stamp: "Formatted at 2025-10-26 15:13:48 on dist-test-slave-8pgz"
server_key: "c8d02ad0fc61c70498df51c2ce766c06"
server_key_iv: "ee0f7ee56a4650f575e9fd26ba676a17"
server_key_version: "encryptionkey@0"
I20251026 15:13:48.356503 2359 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:48.375432 2359 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:48.377503 2359 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:48.377624 2359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:48.377820 2359 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:48.378145 2359 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:48.378180 2359 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:48.378213 2359 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:48.378232 2359 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:48.384204 2359 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:39465
I20251026 15:13:48.384298 2487 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:39465 every 8 connection(s)
I20251026 15:13:48.384583 2359 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:13:48.387122 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 2359
I20251026 15:13:48.387241 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:13:48.387435 15185 external_mini_cluster.cc:1468] Setting key e2fa00fad64bed2eb2f57be8e45c462c
Oct 26 15:13:48 dist-test-slave-8pgz krb5kdc[2111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491628, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.66@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:48.394747 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.385392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:52361 (local address 127.14.212.126:43475)
1026 15:13:48.385475 (+ 83us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.385478 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.385740 (+ 262us) server_negotiation.cc:408] Connection header received
1026 15:13:48.386257 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.386259 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.386269 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.386327 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:48.387274 (+ 947us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.387895 (+ 621us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.388493 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.388660 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.391663 (+ 3003us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:48.391669 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:48.391669 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:48.391679 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:48.393462 (+ 1783us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:48.393847 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:48.393849 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:48.393849 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:48.393876 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:48.394046 (+ 170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:48.394046 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:48.394047 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:48.394179 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:48.394253 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.394532 (+ 279us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.394640 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251026 15:13:48.394785 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.385330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43475 (local address 127.14.212.66:52361)
1026 15:13:48.385522 (+ 192us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:48.385554 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:48.386143 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:48.386386 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:48.386407 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:48.386760 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:48.387121 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.387125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.387991 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.387992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:48.388423 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.388426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.388582 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.389759 (+ 1177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:48.389785 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:48.391541 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:48.393553 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:48.393560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:48.393563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:48.393771 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:48.393937 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:48.393937 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:48.393937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:48.393994 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:48.394241 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:48.394243 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:48.394361 (+ 118us) client_negotiation.cc:769] Sending connection context
1026 15:13:48.394496 (+ 135us) client_negotiation.cc:240] Negotiation successful
1026 15:13:48.394618 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
I20251026 15:13:48.395339 2488 heartbeater.cc:344] Connected to a master server at 127.14.212.126:43475
I20251026 15:13:48.395437 2488 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:48.395646 2488 heartbeater.cc:507] Master 127.14.212.126:43475 requested a full tablet report, sending...
I20251026 15:13:48.396050 2159 ts_manager.cc:194] Registered new tserver with Master: d9a56999ddca4395ad0c4160da91bb93 (127.14.212.66:39465)
WARNING: no policy specified for kudu/127.14.212.67@KRBTEST.COM; defaulting to no policy
I20251026 15:13:48.396550 2159 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.66@KRBTEST.COM'} at 127.14.212.66:52361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:48.433719 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:43475
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:48.513958 2495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:48.514166 2495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:48.514204 2495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:48.515686 2495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:48.515734 2495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:48.515775 2495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:13:48.517419 2495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.212.126:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:48.517665 2495 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:48.517922 2495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:48.520658 2503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:48.520651 2500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:48.520725 2501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:48.520808 2495 server_base.cc:1047] running on GCE node
I20251026 15:13:48.521103 2495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:48.521304 2495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:48.522462 2495 hybrid_clock.cc:648] HybridClock initialized: now 1761491628522444 us; error 29 us; skew 500 ppm
Oct 26 15:13:48 dist-test-slave-8pgz krb5kdc[2111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491628, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:48.525058 2495 init.cc:377] Logged in from keytab as kudu/127.14.212.67@KRBTEST.COM (short username kudu)
I20251026 15:13:48.525619 2495 webserver.cc:492] Webserver started at http://127.14.212.67:38959/ using document root <none> and password file <none>
I20251026 15:13:48.525806 2495 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:48.525847 2495 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:48.525971 2495 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:48.526963 2495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "0ff345f1b2e2457bb7a0699a17cc031c"
format_stamp: "Formatted at 2025-10-26 15:13:48 on dist-test-slave-8pgz"
server_key: "052f3b891f417e9165d500c0a055eb37"
server_key_iv: "cc92a0e7f143cdd38c8a6033cb990292"
server_key_version: "encryptionkey@0"
I20251026 15:13:48.527280 2495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "0ff345f1b2e2457bb7a0699a17cc031c"
format_stamp: "Formatted at 2025-10-26 15:13:48 on dist-test-slave-8pgz"
server_key: "052f3b891f417e9165d500c0a055eb37"
server_key_iv: "cc92a0e7f143cdd38c8a6033cb990292"
server_key_version: "encryptionkey@0"
I20251026 15:13:48.528448 2495 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:13:48.529088 2510 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:48.529242 2495 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:13:48.529313 2495 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "0ff345f1b2e2457bb7a0699a17cc031c"
format_stamp: "Formatted at 2025-10-26 15:13:48 on dist-test-slave-8pgz"
server_key: "052f3b891f417e9165d500c0a055eb37"
server_key_iv: "cc92a0e7f143cdd38c8a6033cb990292"
server_key_version: "encryptionkey@0"
I20251026 15:13:48.529374 2495 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:48.548574 2495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:48.550469 2495 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:48.550607 2495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:48.550827 2495 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:48.551142 2495 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:48.551184 2495 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:48.551221 2495 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:48.551254 2495 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:48.556908 2495 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:38667
I20251026 15:13:48.556968 2623 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:38667 every 8 connection(s)
I20251026 15:13:48.557263 2495 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:13:48.558580 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 2495
I20251026 15:13:48.558689 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:13:48.558880 15185 external_mini_cluster.cc:1468] Setting key 2f0511a3356b54bb4fff2aea8a7fc11d
Oct 26 15:13:48 dist-test-slave-8pgz krb5kdc[2111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491628, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.67@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:13:48.566529 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.557936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:48065 (local address 127.14.212.126:43475)
1026 15:13:48.558021 (+ 85us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.558024 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.558125 (+ 101us) server_negotiation.cc:408] Connection header received
1026 15:13:48.558796 (+ 671us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.558798 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.558812 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.558856 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:48.559935 (+ 1079us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.560476 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.561030 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.561165 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.563559 (+ 2394us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:48.563563 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:48.563563 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:48.563570 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:48.565075 (+ 1505us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:48.565500 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:48.565502 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:48.565502 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:48.565526 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:48.565816 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:48.565817 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:48.565817 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:48.565922 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:48.566009 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.566335 (+ 326us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.566460 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251026 15:13:48.566689 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.557858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:43475 (local address 127.14.212.67:48065)
1026 15:13:48.557997 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:48.558031 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:48.558670 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:48.558923 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:48.558928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:48.559425 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:48.559810 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.559815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.560552 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.560553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:48.560931 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.560933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.561046 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.562040 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:13:48.562056 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:48.563455 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:48.565195 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:48.565201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:48.565204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:48.565423 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:48.565650 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:13:48.565651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:13:48.565651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:13:48.565719 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:13:48.566051 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:48.566056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:48.566190 (+ 134us) client_negotiation.cc:769] Sending connection context
1026 15:13:48.566310 (+ 120us) client_negotiation.cc:240] Negotiation successful
1026 15:13:48.566494 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20251026 15:13:48.567262 2624 heartbeater.cc:344] Connected to a master server at 127.14.212.126:43475
I20251026 15:13:48.567363 2624 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:48.567572 2624 heartbeater.cc:507] Master 127.14.212.126:43475 requested a full tablet report, sending...
I20251026 15:13:48.568011 2159 ts_manager.cc:194] Registered new tserver with Master: 0ff345f1b2e2457bb7a0699a17cc031c (127.14.212.67:38667)
I20251026 15:13:48.568372 2159 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.212.67@KRBTEST.COM'} at 127.14.212.67:48065
I20251026 15:13:48.570533 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:13:48.577070 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.571881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.14.212.126:43475)
1026 15:13:48.571927 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.571929 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.571995 (+ 66us) server_negotiation.cc:408] Connection header received
1026 15:13:48.572034 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.572035 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.572043 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.572088 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:48.572740 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.573220 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.573702 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.573804 (+ 102us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.574844 (+ 1040us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:48.574847 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:48.574847 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:48.574853 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:48.576222 (+ 1369us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:48.576577 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:48.576577 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:48.576578 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:48.576593 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:48.576695 (+ 102us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:48.576695 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:48.576696 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:48.576798 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:48.576870 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.576945 (+ 75us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.577007 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251026 15:13:48.578528 2159 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:13:48.579731 2159 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:13:48.584537 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.581747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:38667 (local address 127.0.0.1:55602)
1026 15:13:48.581899 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:48.581925 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:48.582035 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:48.582336 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:48.582339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:48.582341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:48.582631 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.582633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.583644 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.583645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:48.584237 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.584238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.584304 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.584346 (+ 42us) client_negotiation.cc:769] Sending connection context
1026 15:13:48.584405 (+ 59us) client_negotiation.cc:240] Negotiation successful
1026 15:13:48.584446 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":49,"threads_started":1}
I20251026 15:13:48.585067 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.581748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:39465 (local address 127.0.0.1:51518)
1026 15:13:48.582057 (+ 309us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:48.582067 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:48.582128 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:48.582563 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:48.582564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:48.582565 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:48.582775 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.582777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.583972 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.583972 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:48.584900 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.584902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.584946 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.584949 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:13:48.584973 (+ 24us) client_negotiation.cc:240] Negotiation successful
1026 15:13:48.585005 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":5,"thread_start_us":89,"threads_started":1}
I20251026 15:13:48.585114 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.581839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.14.212.67:38667)
1026 15:13:48.582173 (+ 334us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.582183 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.582194 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:13:48.582233 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.582235 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.582252 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.582324 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:48.582721 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.583570 (+ 849us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.584336 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.584951 (+ 615us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.585007 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.585031 (+ 24us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.585060 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":135,"threads_started":1}
I20251026 15:13:48.585373 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.581930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:43679 (local address 127.0.0.1:42942)
1026 15:13:48.582357 (+ 427us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:48.582366 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:48.582459 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:48.582882 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:48.582883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:48.582884 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:48.583070 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.583071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.584342 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.584343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:48.585167 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.585169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.585240 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.585244 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:48.585281 (+ 37us) client_negotiation.cc:240] Negotiation successful
1026 15:13:48.585323 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":70,"threads_started":1}
I20251026 15:13:48.585614 2558 tablet_service.cc:1505] Processing CreateTablet for tablet a3547ef33efa4a9b8386d014cd260d28 (DEFAULT_TABLE table=test-table [id=58a0aa6f6529450d86401b11a40ccc42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:48.585878 2558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3547ef33efa4a9b8386d014cd260d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:48.586275 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.582144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.14.212.65:43679)
1026 15:13:48.582307 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.582315 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.582441 (+ 126us) server_negotiation.cc:408] Connection header received
1026 15:13:48.582535 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.582537 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.582554 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.582621 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:48.583150 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.584256 (+ 1106us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.585531 (+ 1275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.586087 (+ 556us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.586134 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.586163 (+ 29us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.586200 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20251026 15:13:48.586382 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.581909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51518 (local address 127.14.212.66:39465)
1026 15:13:48.582173 (+ 264us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.582183 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.582194 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:13:48.582241 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.582244 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.582259 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.582712 (+ 453us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:48.582849 (+ 137us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.583872 (+ 1023us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.585672 (+ 1800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.586182 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.586231 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.586258 (+ 27us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.586302 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
I20251026 15:13:48.586782 2286 tablet_service.cc:1505] Processing CreateTablet for tablet a3547ef33efa4a9b8386d014cd260d28 (DEFAULT_TABLE table=test-table [id=58a0aa6f6529450d86401b11a40ccc42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:48.586796 2422 tablet_service.cc:1505] Processing CreateTablet for tablet a3547ef33efa4a9b8386d014cd260d28 (DEFAULT_TABLE table=test-table [id=58a0aa6f6529450d86401b11a40ccc42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:48.587061 2422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3547ef33efa4a9b8386d014cd260d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:48.587061 2286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3547ef33efa4a9b8386d014cd260d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:48.588201 2641 tablet_bootstrap.cc:492] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c: Bootstrap starting.
I20251026 15:13:48.588851 2641 tablet_bootstrap.cc:654] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:48.589116 2641 log.cc:826] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:48.589740 2644 tablet_bootstrap.cc:492] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce: Bootstrap starting.
I20251026 15:13:48.589774 2641 tablet_bootstrap.cc:492] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c: No bootstrap required, opened a new log
I20251026 15:13:48.589828 2641 ts_tablet_manager.cc:1403] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:13:48.590459 2644 tablet_bootstrap.cc:654] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:48.590597 2643 tablet_bootstrap.cc:492] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93: Bootstrap starting.
I20251026 15:13:48.590740 2644 log.cc:826] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:48.591159 2643 tablet_bootstrap.cc:654] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:48.591501 2643 log.cc:826] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:48.591593 2644 tablet_bootstrap.cc:492] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce: No bootstrap required, opened a new log
I20251026 15:13:48.591511 2641 raft_consensus.cc:359] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38667 } } peers { permanent_uuid: "d9a56999ddca4395ad0c4160da91bb93" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 39465 } } peers { permanent_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43679 } }
I20251026 15:13:48.591650 2641 raft_consensus.cc:385] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:48.591655 2644 ts_tablet_manager.cc:1403] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 15:13:48.591688 2641 raft_consensus.cc:740] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff345f1b2e2457bb7a0699a17cc031c, State: Initialized, Role: FOLLOWER
I20251026 15:13:48.591807 2641 consensus_queue.cc:260] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38667 } } peers { permanent_uuid: "d9a56999ddca4395ad0c4160da91bb93" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 39465 } } peers { permanent_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43679 } }
I20251026 15:13:48.592079 2641 ts_tablet_manager.cc:1434] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:13:48.592115 2624 heartbeater.cc:499] Master 127.14.212.126:43475 was elected leader, sending a full tablet report...
I20251026 15:13:48.592145 2643 tablet_bootstrap.cc:492] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93: No bootstrap required, opened a new log
I20251026 15:13:48.592193 2643 ts_tablet_manager.cc:1403] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:13:48.592834 2644 raft_consensus.cc:359] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38667 } } peers { permanent_uuid: "d9a56999ddca4395ad0c4160da91bb93" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 39465 } } peers { permanent_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43679 } }
I20251026 15:13:48.592927 2644 raft_consensus.cc:385] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:48.592945 2644 raft_consensus.cc:740] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2c4913e51b4baa985f7ae9126d42ce, State: Initialized, Role: FOLLOWER
I20251026 15:13:48.593017 2644 consensus_queue.cc:260] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38667 } } peers { permanent_uuid: "d9a56999ddca4395ad0c4160da91bb93" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 39465 } } peers { permanent_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43679 } }
I20251026 15:13:48.593245 2644 ts_tablet_manager.cc:1434] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 15:13:48.593250 2352 heartbeater.cc:499] Master 127.14.212.126:43475 was elected leader, sending a full tablet report...
I20251026 15:13:48.593897 2643 raft_consensus.cc:359] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38667 } } peers { permanent_uuid: "d9a56999ddca4395ad0c4160da91bb93" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 39465 } } peers { permanent_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43679 } }
I20251026 15:13:48.594041 2643 raft_consensus.cc:385] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:48.594070 2643 raft_consensus.cc:740] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a56999ddca4395ad0c4160da91bb93, State: Initialized, Role: FOLLOWER
I20251026 15:13:48.594151 2643 consensus_queue.cc:260] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38667 } } peers { permanent_uuid: "d9a56999ddca4395ad0c4160da91bb93" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 39465 } } peers { permanent_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43679 } }
I20251026 15:13:48.594383 2643 ts_tablet_manager.cc:1434] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:13:48.594441 2488 heartbeater.cc:499] Master 127.14.212.126:43475 was elected leader, sending a full tablet report...
W20251026 15:13:48.635109 2489 tablet.cc:2378] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 15:13:48.684150 2647 raft_consensus.cc:493] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:13:48.684291 2647 raft_consensus.cc:515] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38667 } } peers { permanent_uuid: "d9a56999ddca4395ad0c4160da91bb93" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 39465 } } peers { permanent_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43679 } }
I20251026 15:13:48.684639 2647 leader_election.cc:290] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9a56999ddca4395ad0c4160da91bb93 (127.14.212.66:39465), 6c2c4913e51b4baa985f7ae9126d42ce (127.14.212.65:43679)
I20251026 15:13:48.686234 2648 raft_consensus.cc:493] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:13:48.686334 2648 raft_consensus.cc:515] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38667 } } peers { permanent_uuid: "d9a56999ddca4395ad0c4160da91bb93" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 39465 } } peers { permanent_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43679 } }
I20251026 15:13:48.686663 2648 leader_election.cc:290] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ff345f1b2e2457bb7a0699a17cc031c (127.14.212.67:38667), d9a56999ddca4395ad0c4160da91bb93 (127.14.212.66:39465)
I20251026 15:13:48.687081 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.684822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:43679 (local address 127.14.212.67:58957)
1026 15:13:48.684876 (+ 54us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:48.684890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:48.684942 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:48.685144 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:48.685145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:48.685146 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:48.685335 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.685336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.685977 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.685978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:48.686841 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.686844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.686911 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.686915 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:48.686946 (+ 31us) client_negotiation.cc:240] Negotiation successful
1026 15:13:48.686997 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20251026 15:13:48.687789 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.684915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:52309 (local address 127.14.212.66:39465)
1026 15:13:48.684971 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.684974 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.685127 (+ 153us) server_negotiation.cc:408] Connection header received
1026 15:13:48.685199 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.685201 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.685212 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.685262 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:48.685576 (+ 314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.686049 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.687077 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.687593 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.687602 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.687675 (+ 73us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.687726 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251026 15:13:48.687858 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.684953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:58957 (local address 127.14.212.65:43679)
1026 15:13:48.685029 (+ 76us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.685031 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.685041 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:13:48.685066 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.685066 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.685075 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.685124 (+ 49us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:48.685415 (+ 291us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.685889 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.687238 (+ 1349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.687714 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.687721 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.687745 (+ 24us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.687779 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251026 15:13:48.688107 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.684824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:39465 (local address 127.14.212.67:52309)
1026 15:13:48.685041 (+ 217us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:48.685050 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:48.685118 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:48.685273 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:48.685274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:48.685275 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:48.685501 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.685502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.686122 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.686123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:48.686984 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.686986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.687587 (+ 601us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.687591 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:48.687994 (+ 403us) client_negotiation.cc:240] Negotiation successful
1026 15:13:48.688034 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":11,"thread_start_us":137,"threads_started":1}
I20251026 15:13:48.688144 2306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3547ef33efa4a9b8386d014cd260d28" candidate_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" is_pre_election: true
I20251026 15:13:48.688246 2306 raft_consensus.cc:2468] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ff345f1b2e2457bb7a0699a17cc031c in term 0.
I20251026 15:13:48.688455 2513 leader_election.cc:304] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff345f1b2e2457bb7a0699a17cc031c, 6c2c4913e51b4baa985f7ae9126d42ce; no voters:
I20251026 15:13:48.688513 2442 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3547ef33efa4a9b8386d014cd260d28" candidate_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a56999ddca4395ad0c4160da91bb93" is_pre_election: true
I20251026 15:13:48.688555 2647 raft_consensus.cc:2804] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:13:48.688609 2647 raft_consensus.cc:493] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:13:48.688627 2647 raft_consensus.cc:3060] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:48.688634 2442 raft_consensus.cc:2468] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ff345f1b2e2457bb7a0699a17cc031c in term 0.
I20251026 15:13:48.689360 2647 raft_consensus.cc:515] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38667 } } peers { permanent_uuid: "d9a56999ddca4395ad0c4160da91bb93" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 39465 } } peers { permanent_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43679 } }
I20251026 15:13:48.689491 2647 leader_election.cc:290] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [CANDIDATE]: Term 1 election: Requested vote from peers d9a56999ddca4395ad0c4160da91bb93 (127.14.212.66:39465), 6c2c4913e51b4baa985f7ae9126d42ce (127.14.212.65:43679)
I20251026 15:13:48.689491 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.686781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:38667 (local address 127.14.212.65:37177)
1026 15:13:48.686867 (+ 86us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:48.686877 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:48.686915 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:48.687261 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:48.687262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:48.687281 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:48.687489 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.687491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.688403 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.688404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:48.689265 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.689268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.689357 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.689360 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:13:48.689393 (+ 33us) client_negotiation.cc:240] Negotiation successful
1026 15:13:48.689429 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20251026 15:13:48.689733 2442 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3547ef33efa4a9b8386d014cd260d28" candidate_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a56999ddca4395ad0c4160da91bb93"
I20251026 15:13:48.689806 2442 raft_consensus.cc:3060] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:48.689797 2306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3547ef33efa4a9b8386d014cd260d28" candidate_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2c4913e51b4baa985f7ae9126d42ce"
I20251026 15:13:48.689872 2306 raft_consensus.cc:3060] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:48.689937 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.686990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:37177 (local address 127.14.212.67:38667)
1026 15:13:48.687027 (+ 37us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.687030 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.687037 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:13:48.687057 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.687057 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.687066 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.687114 (+ 48us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:48.687573 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.688293 (+ 720us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.689368 (+ 1075us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.689826 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.689833 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.689853 (+ 20us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.689886 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251026 15:13:48.690173 2578 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3547ef33efa4a9b8386d014cd260d28" candidate_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" is_pre_election: true
I20251026 15:13:48.690276 2578 raft_consensus.cc:2393] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6c2c4913e51b4baa985f7ae9126d42ce in current term 1: Already voted for candidate 0ff345f1b2e2457bb7a0699a17cc031c in this term.
I20251026 15:13:48.690774 2306 raft_consensus.cc:2468] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ff345f1b2e2457bb7a0699a17cc031c in term 1.
I20251026 15:13:48.690781 2442 raft_consensus.cc:2468] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ff345f1b2e2457bb7a0699a17cc031c in term 1.
I20251026 15:13:48.690951 2513 leader_election.cc:304] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff345f1b2e2457bb7a0699a17cc031c, 6c2c4913e51b4baa985f7ae9126d42ce; no voters:
I20251026 15:13:48.691042 2647 raft_consensus.cc:2804] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:48.691190 2647 raft_consensus.cc:697] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 1 LEADER]: Becoming Leader. State: Replica: 0ff345f1b2e2457bb7a0699a17cc031c, State: Running, Role: LEADER
I20251026 15:13:48.691295 2647 consensus_queue.cc:237] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38667 } } peers { permanent_uuid: "d9a56999ddca4395ad0c4160da91bb93" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 39465 } } peers { permanent_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43679 } }
I20251026 15:13:48.691597 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.686841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:39465 (local address 127.14.212.65:42069)
1026 15:13:48.687218 (+ 377us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:48.687228 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:48.687274 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:48.687941 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:48.687942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:48.687944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:48.688206 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.688208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.690375 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.690376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:48.691376 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:48.691381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.691453 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.691457 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:48.691485 (+ 28us) client_negotiation.cc:240] Negotiation successful
1026 15:13:48.691522 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":109,"threads_started":1}
I20251026 15:13:48.692194 2159 catalog_manager.cc:5649] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ff345f1b2e2457bb7a0699a17cc031c (127.14.212.67). New cstate: current_term: 1 leader_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff345f1b2e2457bb7a0699a17cc031c" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 38667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9a56999ddca4395ad0c4160da91bb93" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 39465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43679 } health_report { overall_health: UNKNOWN } } }
I20251026 15:13:48.692350 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.687050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:42069 (local address 127.14.212.66:39465)
1026 15:13:48.687723 (+ 673us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.687725 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.687736 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:13:48.687755 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.687756 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.687766 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.687831 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:48.688318 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.690171 (+ 1853us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.691690 (+ 1519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.692223 (+ 533us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.692232 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.692257 (+ 25us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.692288 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":90,"threads_started":1}
I20251026 15:13:48.692543 2442 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3547ef33efa4a9b8386d014cd260d28" candidate_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a56999ddca4395ad0c4160da91bb93" is_pre_election: true
I20251026 15:13:48.692633 2442 raft_consensus.cc:2393] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6c2c4913e51b4baa985f7ae9126d42ce in current term 1: Already voted for candidate 0ff345f1b2e2457bb7a0699a17cc031c in this term.
I20251026 15:13:48.692878 2239 leader_election.cc:304] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2c4913e51b4baa985f7ae9126d42ce; no voters: 0ff345f1b2e2457bb7a0699a17cc031c, d9a56999ddca4395ad0c4160da91bb93
I20251026 15:13:48.693017 2648 raft_consensus.cc:2749] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 15:13:48.702997 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.700937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.14.212.67:38667)
1026 15:13:48.701000 (+ 63us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.701002 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.701011 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:13:48.701052 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.701053 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.701069 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.701123 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:13:48.701370 (+ 247us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.701826 (+ 456us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.702494 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.702634 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.702668 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:13:48.702757 (+ 89us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:13:48.702829 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.702881 (+ 52us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.702914 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251026 15:13:48.704679 2306 raft_consensus.cc:1275] T a3547ef33efa4a9b8386d014cd260d28 P 6c2c4913e51b4baa985f7ae9126d42ce [term 1 FOLLOWER]: Refusing update from remote peer 0ff345f1b2e2457bb7a0699a17cc031c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:48.704664 2442 raft_consensus.cc:1275] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93 [term 1 FOLLOWER]: Refusing update from remote peer 0ff345f1b2e2457bb7a0699a17cc031c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:48.705029 2653 consensus_queue.cc:1048] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a56999ddca4395ad0c4160da91bb93" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:48.705154 2647 consensus_queue.cc:1048] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2c4913e51b4baa985f7ae9126d42ce" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:48.707520 2657 mvcc.cc:204] Tried to move back new op lower bound from 7215069711171829760 to 7215069711120162816. Current Snapshot: MvccSnapshot[applied={T|T < 7215069711171829760}]
I20251026 15:13:48.707733 2658 mvcc.cc:204] Tried to move back new op lower bound from 7215069711171829760 to 7215069711120162816. Current Snapshot: MvccSnapshot[applied={T|T < 7215069711171829760}]
I20251026 15:13:48.710274 2158 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34492:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:48.710357 2158 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34492:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:48.711133 2158 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8acd68391a784cb5943fa6d1aaa5afc9: Sending DeleteTablet for 3 replicas of tablet a3547ef33efa4a9b8386d014cd260d28
I20251026 15:13:48.711452 2422 tablet_service.cc:1552] Processing DeleteTablet for tablet a3547ef33efa4a9b8386d014cd260d28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:48 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:51518
I20251026 15:13:48.711460 2286 tablet_service.cc:1552] Processing DeleteTablet for tablet a3547ef33efa4a9b8386d014cd260d28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:48 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:42942
I20251026 15:13:48.711719 2558 tablet_service.cc:1552] Processing DeleteTablet for tablet a3547ef33efa4a9b8386d014cd260d28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:48 UTC) from {username='kudu', principal='kudu/127.14.212.126@KRBTEST.COM'} at 127.0.0.1:55602
I20251026 15:13:48.711941 2668 tablet_replica.cc:333] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c: stopping tablet replica
I20251026 15:13:48.712035 2668 raft_consensus.cc:2243] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:13:48.712141 2668 raft_consensus.cc:2272] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:48.712224 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 2223
I20251026 15:13:48.715076 2668 ts_tablet_manager.cc:1916] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:48.716791 2668 ts_tablet_manager.cc:1929] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:48.716871 2668 log.cc:1199] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/a3547ef33efa4a9b8386d014cd260d28
I20251026 15:13:48.717152 2668 ts_tablet_manager.cc:1950] T a3547ef33efa4a9b8386d014cd260d28 P 0ff345f1b2e2457bb7a0699a17cc031c: Deleting consensus metadata
I20251026 15:13:48.717533 2145 catalog_manager.cc:4985] TS 0ff345f1b2e2457bb7a0699a17cc031c (127.14.212.67:38667): tablet a3547ef33efa4a9b8386d014cd260d28 (table test-table [id=58a0aa6f6529450d86401b11a40ccc42]) successfully deleted
I20251026 15:13:48.718509 2666 tablet_replica.cc:333] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93: stopping tablet replica
I20251026 15:13:48.718607 2666 raft_consensus.cc:2243] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:48.718722 2666 raft_consensus.cc:2272] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:48.719153 2666 ts_tablet_manager.cc:1916] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:48.721189 2666 ts_tablet_manager.cc:1929] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:48.721266 2666 log.cc:1199] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/a3547ef33efa4a9b8386d014cd260d28
I20251026 15:13:48.721536 2666 ts_tablet_manager.cc:1950] T a3547ef33efa4a9b8386d014cd260d28 P d9a56999ddca4395ad0c4160da91bb93: Deleting consensus metadata
I20251026 15:13:48.722008 2143 catalog_manager.cc:4985] TS d9a56999ddca4395ad0c4160da91bb93 (127.14.212.66:39465): tablet a3547ef33efa4a9b8386d014cd260d28 (table test-table [id=58a0aa6f6529450d86401b11a40ccc42]) successfully deleted
W20251026 15:13:48.723073 2145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:43679 (error 108)
I20251026 15:13:48.723387 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 2359
I20251026 15:13:48.723995 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.723495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:43679 (local address 127.0.0.1:42954)
1026 15:13:48.723920 (+ 425us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:48.723957 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:43679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404}
W20251026 15:13:48.725713 2145 catalog_manager.cc:4712] TS 6c2c4913e51b4baa985f7ae9126d42ce (127.14.212.65:43679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3547ef33efa4a9b8386d014cd260d28: Network error: Client connection negotiation failed: client connection to 127.14.212.65:43679: connect: Connection refused (error 111)
I20251026 15:13:48.728988 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 2495
I20251026 15:13:48.733264 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 2127
2025-10-26T15:13:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2982 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2025-10-26T15:13:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:13:48Z Disabled control of system clock
I20251026 15:13:48.760258 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46785
--webserver_interface=127.14.212.126
--webserver_port=0
--builtin_ntp_servers=127.14.212.84:33085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:46785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251026 15:13:48.841239 2673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:48.841444 2673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:48.841468 2673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:48.843000 2673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:13:48.843050 2673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:48.843062 2673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:48.843075 2673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:13:48.843086 2673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:13:48.844763 2673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:46785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:48.844973 2673 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:48.845171 2673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:48.848245 2679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:48.848291 2673 server_base.cc:1047] running on GCE node
W20251026 15:13:48.848359 2681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:48.848364 2678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:48.848711 2673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:48.848949 2673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:48.850190 2673 hybrid_clock.cc:648] HybridClock initialized: now 1761491628850150 us; error 51 us; skew 500 ppm
I20251026 15:13:48.851562 2673 webserver.cc:492] Webserver started at http://127.14.212.126:42277/ using document root <none> and password file <none>
I20251026 15:13:48.851754 2673 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:48.851794 2673 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:48.851881 2673 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:48.852836 2673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "d657eeee047e4e20ac9278a12c5dfca5"
format_stamp: "Formatted at 2025-10-26 15:13:48 on dist-test-slave-8pgz"
server_key: "69f11316a0d5da35e4c8e8d20a42c7b2"
server_key_iv: "3b18da8c791e82a20ab47d5dc5acfe73"
server_key_version: "encryptionkey@0"
I20251026 15:13:48.853127 2673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "d657eeee047e4e20ac9278a12c5dfca5"
format_stamp: "Formatted at 2025-10-26 15:13:48 on dist-test-slave-8pgz"
server_key: "69f11316a0d5da35e4c8e8d20a42c7b2"
server_key_iv: "3b18da8c791e82a20ab47d5dc5acfe73"
server_key_version: "encryptionkey@0"
I20251026 15:13:48.854458 2673 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:13:48.855368 2687 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:48.855634 2673 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:48.855720 2673 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "d657eeee047e4e20ac9278a12c5dfca5"
format_stamp: "Formatted at 2025-10-26 15:13:48 on dist-test-slave-8pgz"
server_key: "69f11316a0d5da35e4c8e8d20a42c7b2"
server_key_iv: "3b18da8c791e82a20ab47d5dc5acfe73"
server_key_version: "encryptionkey@0"
I20251026 15:13:48.855792 2673 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:48.878669 2673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:48.878978 2673 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:48.879107 2673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:48.883054 2673 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:46785
I20251026 15:13:48.883100 2739 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:46785 every 8 connection(s)
I20251026 15:13:48.883421 2673 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:13:48.884073 2740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:48.884438 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 2673
I20251026 15:13:48.884546 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:13:48.884747 15185 external_mini_cluster.cc:1468] Setting key 43db393c8afff01fcee2c2f82068ed98
I20251026 15:13:48.887130 2740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5: Bootstrap starting.
I20251026 15:13:48.887804 2740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:48.888119 2740 log.cc:826] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:48.888976 2740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5: No bootstrap required, opened a new log
I20251026 15:13:48.890055 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:48.885480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.14.212.126:46785)
1026 15:13:48.885709 (+ 229us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:48.885718 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:48.885743 (+ 25us) server_negotiation.cc:408] Connection header received
1026 15:13:48.886161 (+ 418us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:48.886169 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:48.886240 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:48.886421 (+ 181us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:48.886832 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.887993 (+ 1161us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:48.888764 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:48.889013 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:48.889393 (+ 380us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:48.889400 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:48.889401 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:48.889609 (+ 208us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:48.889643 (+ 34us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:48.889652 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:48.889736 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:48.889824 (+ 88us) server_negotiation.cc:300] Negotiation successful
1026 15:13:48.889882 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20251026 15:13:48.890566 2740 raft_consensus.cc:359] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d657eeee047e4e20ac9278a12c5dfca5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46785 } }
I20251026 15:13:48.890713 2740 raft_consensus.cc:385] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:48.890738 2740 raft_consensus.cc:740] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d657eeee047e4e20ac9278a12c5dfca5, State: Initialized, Role: FOLLOWER
I20251026 15:13:48.890822 2740 consensus_queue.cc:260] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d657eeee047e4e20ac9278a12c5dfca5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46785 } }
I20251026 15:13:48.890902 2740 raft_consensus.cc:399] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:13:48.890945 2740 raft_consensus.cc:493] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:13:48.890995 2740 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:48.891570 2740 raft_consensus.cc:515] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d657eeee047e4e20ac9278a12c5dfca5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46785 } }
I20251026 15:13:48.891686 2740 leader_election.cc:304] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d657eeee047e4e20ac9278a12c5dfca5; no voters:
I20251026 15:13:48.891839 2740 leader_election.cc:290] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:13:48.891922 2745 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:48.892086 2745 raft_consensus.cc:697] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [term 1 LEADER]: Becoming Leader. State: Replica: d657eeee047e4e20ac9278a12c5dfca5, State: Running, Role: LEADER
I20251026 15:13:48.892086 2740 sys_catalog.cc:565] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:13:48.892194 2745 consensus_queue.cc:237] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d657eeee047e4e20ac9278a12c5dfca5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46785 } }
I20251026 15:13:48.892570 2746 sys_catalog.cc:455] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d657eeee047e4e20ac9278a12c5dfca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d657eeee047e4e20ac9278a12c5dfca5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46785 } } }
I20251026 15:13:48.892582 2747 sys_catalog.cc:455] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d657eeee047e4e20ac9278a12c5dfca5. Latest consensus state: current_term: 1 leader_uuid: "d657eeee047e4e20ac9278a12c5dfca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d657eeee047e4e20ac9278a12c5dfca5" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46785 } } }
I20251026 15:13:48.892740 2747 sys_catalog.cc:458] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:48.892694 2746 sys_catalog.cc:458] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:48.892951 2754 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:13:48.893493 2754 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:13:48.895109 2754 catalog_manager.cc:1357] Generated new cluster ID: ebb47cb385d043ff9724d9ff66f14694
I20251026 15:13:48.895174 2754 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:13:48.906845 2754 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:13:48.907413 2754 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:13:48.910518 2754 catalog_manager.cc:6022] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5: Generated new TSK 0
I20251026 15:13:48.910730 2754 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:13:48.913096 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.65:0
--local_ip_for_outbound_sockets=127.14.212.65
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46785
--builtin_ntp_servers=127.14.212.84:33085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251026 15:13:49.002007 2764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:49.002200 2764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:49.002223 2764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:49.003765 2764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:49.003820 2764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:49.003846 2764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.65
I20251026 15:13:49.005429 2764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.212.65
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.65
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:49.005725 2764 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:49.005961 2764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:49.008760 2769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:49.008929 2770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:49.008778 2772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:49.009150 2764 server_base.cc:1047] running on GCE node
I20251026 15:13:49.009328 2764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:49.009534 2764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:49.010675 2764 hybrid_clock.cc:648] HybridClock initialized: now 1761491629010663 us; error 32 us; skew 500 ppm
I20251026 15:13:49.011863 2764 webserver.cc:492] Webserver started at http://127.14.212.65:41927/ using document root <none> and password file <none>
I20251026 15:13:49.012094 2764 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:49.012141 2764 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:49.012259 2764 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:49.013228 2764 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/data/instance:
uuid: "c65970c705294f9087444d359228e13f"
format_stamp: "Formatted at 2025-10-26 15:13:49 on dist-test-slave-8pgz"
server_key: "124a4e23a86ef03a47f04c0a618d9996"
server_key_iv: "453f7f4b57e873749a3838a4eb7f033d"
server_key_version: "encryptionkey@0"
I20251026 15:13:49.013540 2764 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance:
uuid: "c65970c705294f9087444d359228e13f"
format_stamp: "Formatted at 2025-10-26 15:13:49 on dist-test-slave-8pgz"
server_key: "124a4e23a86ef03a47f04c0a618d9996"
server_key_iv: "453f7f4b57e873749a3838a4eb7f033d"
server_key_version: "encryptionkey@0"
I20251026 15:13:49.014811 2764 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:13:49.015548 2778 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:49.015738 2764 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:49.015817 2764 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/wal
uuid: "c65970c705294f9087444d359228e13f"
format_stamp: "Formatted at 2025-10-26 15:13:49 on dist-test-slave-8pgz"
server_key: "124a4e23a86ef03a47f04c0a618d9996"
server_key_iv: "453f7f4b57e873749a3838a4eb7f033d"
server_key_version: "encryptionkey@0"
I20251026 15:13:49.015884 2764 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:49.025853 2764 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:49.026170 2764 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:49.026293 2764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:49.026546 2764 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:49.026847 2764 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:49.026880 2764 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:49.026917 2764 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:49.026945 2764 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:49.033851 2764 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.65:36637
I20251026 15:13:49.033922 2891 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.65:36637 every 8 connection(s)
I20251026 15:13:49.034216 2764 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/data/info.pb
I20251026 15:13:49.038076 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 2764
I20251026 15:13:49.038189 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-0/wal/instance
I20251026 15:13:49.038424 15185 external_mini_cluster.cc:1468] Setting key 386064098244da106dda66204ba7b3bc
I20251026 15:13:49.038731 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.034947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.65:37721 (local address 127.14.212.126:46785)
1026 15:13:49.035014 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:49.035016 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:49.035224 (+ 208us) server_negotiation.cc:408] Connection header received
1026 15:13:49.035784 (+ 560us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:49.035785 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:49.035801 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:49.035862 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:49.036313 (+ 451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.036825 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.037377 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.037550 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.038343 (+ 793us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:49.038347 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:49.038347 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:49.038352 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:49.038371 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:49.038375 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:49.038461 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:49.038607 (+ 146us) server_negotiation.cc:300] Negotiation successful
1026 15:13:49.038652 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251026 15:13:49.038841 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.034923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46785 (local address 127.14.212.65:37721)
1026 15:13:49.035119 (+ 196us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:49.035143 (+ 24us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:49.035690 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:49.035921 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:49.035924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:49.035933 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:49.036232 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.036234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.036914 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.036915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:49.037305 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.037307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.037438 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.038170 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:13:49.038171 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:49.038224 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:13:49.038249 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:49.038491 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:49.038497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:49.038500 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:13:49.038600 (+ 100us) client_negotiation.cc:240] Negotiation successful
1026 15:13:49.038681 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20251026 15:13:49.039340 2892 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46785
I20251026 15:13:49.039430 2892 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:49.039445 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.66:0
--local_ip_for_outbound_sockets=127.14.212.66
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46785
--builtin_ntp_servers=127.14.212.84:33085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251026 15:13:49.039644 2892 heartbeater.cc:507] Master 127.14.212.126:46785 requested a full tablet report, sending...
I20251026 15:13:49.040102 2704 ts_manager.cc:194] Registered new tserver with Master: c65970c705294f9087444d359228e13f (127.14.212.65:36637)
I20251026 15:13:49.041038 2704 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.14.212.65:37721
W20251026 15:13:49.118273 2895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:49.118501 2895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:49.118532 2895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:49.120040 2895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:49.120081 2895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:49.120106 2895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.66
I20251026 15:13:49.121560 2895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.212.66
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.66
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:49.121811 2895 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:49.122057 2895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:49.124630 2903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:49.124696 2901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:49.124612 2900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:49.124881 2895 server_base.cc:1047] running on GCE node
I20251026 15:13:49.125020 2895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:49.125190 2895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:49.126323 2895 hybrid_clock.cc:648] HybridClock initialized: now 1761491629126310 us; error 28 us; skew 500 ppm
I20251026 15:13:49.127441 2895 webserver.cc:492] Webserver started at http://127.14.212.66:40797/ using document root <none> and password file <none>
I20251026 15:13:49.127619 2895 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:49.127655 2895 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:49.127746 2895 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:49.128674 2895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/data/instance:
uuid: "7fa088e064c24a328d23a0849f618273"
format_stamp: "Formatted at 2025-10-26 15:13:49 on dist-test-slave-8pgz"
server_key: "7f7268a567fd5b07691c25d2c5773071"
server_key_iv: "1eb5dc8b7b1cc1f8fe71a76aec7d6556"
server_key_version: "encryptionkey@0"
I20251026 15:13:49.128938 2895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance:
uuid: "7fa088e064c24a328d23a0849f618273"
format_stamp: "Formatted at 2025-10-26 15:13:49 on dist-test-slave-8pgz"
server_key: "7f7268a567fd5b07691c25d2c5773071"
server_key_iv: "1eb5dc8b7b1cc1f8fe71a76aec7d6556"
server_key_version: "encryptionkey@0"
I20251026 15:13:49.130098 2895 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:13:49.130900 2909 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:49.131105 2895 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251026 15:13:49.131189 2895 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/wal
uuid: "7fa088e064c24a328d23a0849f618273"
format_stamp: "Formatted at 2025-10-26 15:13:49 on dist-test-slave-8pgz"
server_key: "7f7268a567fd5b07691c25d2c5773071"
server_key_iv: "1eb5dc8b7b1cc1f8fe71a76aec7d6556"
server_key_version: "encryptionkey@0"
I20251026 15:13:49.131243 2895 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:49.143878 2895 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:49.144212 2895 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:49.144315 2895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:49.144486 2895 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:49.144788 2895 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:49.144820 2895 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:49.144843 2895 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:49.144870 2895 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:49.150563 2895 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.66:36173
I20251026 15:13:49.150616 3022 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.66:36173 every 8 connection(s)
I20251026 15:13:49.150918 2895 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/data/info.pb
I20251026 15:13:49.154351 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 2895
I20251026 15:13:49.154482 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/wal/instance
I20251026 15:13:49.154711 15185 external_mini_cluster.cc:1468] Setting key 5558428f4dd7712d43360ff8ef5d1a5b
I20251026 15:13:49.155202 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.151610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:38073 (local address 127.14.212.126:46785)
1026 15:13:49.151676 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:49.151678 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:49.151860 (+ 182us) server_negotiation.cc:408] Connection header received
1026 15:13:49.152370 (+ 510us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:49.152372 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:49.152388 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:49.152448 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:49.152904 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.153392 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.153938 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.154101 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.154889 (+ 788us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:49.154894 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:49.154894 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:49.154900 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:49.154916 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:49.154920 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:49.154979 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:49.155062 (+ 83us) server_negotiation.cc:300] Negotiation successful
1026 15:13:49.155097 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251026 15:13:49.155459 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.151580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46785 (local address 127.14.212.66:38073)
1026 15:13:49.151758 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:49.151783 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:49.152274 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:49.152511 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:49.152515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:49.152523 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:49.152817 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.152820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.153472 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.153475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:49.153862 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.153864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.154006 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.154702 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:13:49.154702 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:49.154731 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:13:49.154753 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:49.154995 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:49.155000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:49.155002 (+ 2us) client_negotiation.cc:769] Sending connection context
1026 15:13:49.155067 (+ 65us) client_negotiation.cc:240] Negotiation successful
1026 15:13:49.155229 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20251026 15:13:49.155876 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.212.67:0
--local_ip_for_outbound_sockets=127.14.212.67
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46785
--builtin_ntp_servers=127.14.212.84:33085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251026 15:13:49.155959 3023 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46785
I20251026 15:13:49.156054 3023 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:49.156270 3023 heartbeater.cc:507] Master 127.14.212.126:46785 requested a full tablet report, sending...
I20251026 15:13:49.156656 2704 ts_manager.cc:194] Registered new tserver with Master: 7fa088e064c24a328d23a0849f618273 (127.14.212.66:36173)
I20251026 15:13:49.157068 2704 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.14.212.66:38073
W20251026 15:13:49.236189 3026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:49.236375 3026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:49.236397 3026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:49.237826 3026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:49.237874 3026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:49.237905 3026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.212.67
I20251026 15:13:49.239432 3026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.212.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.212.67
--webserver_port=0
--tserver_master_addrs=127.14.212.126:46785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.212.67
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:49.239652 3026 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:49.239867 3026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:49.242640 3034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:49.242591 3031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:49.242594 3032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:49.242947 3026 server_base.cc:1047] running on GCE node
I20251026 15:13:49.243139 3026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:49.243355 3026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:49.244511 3026 hybrid_clock.cc:648] HybridClock initialized: now 1761491629244498 us; error 31 us; skew 500 ppm
I20251026 15:13:49.245733 3026 webserver.cc:492] Webserver started at http://127.14.212.67:44149/ using document root <none> and password file <none>
I20251026 15:13:49.245939 3026 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:49.245990 3026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:49.246110 3026 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:49.247123 3026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/data/instance:
uuid: "e3049ca6595d424f88b45463dc85629f"
format_stamp: "Formatted at 2025-10-26 15:13:49 on dist-test-slave-8pgz"
server_key: "495c9a2ba251c7b91520bce2fce83eb3"
server_key_iv: "20be0cfce28013438f9694b1be0d51ba"
server_key_version: "encryptionkey@0"
I20251026 15:13:49.247423 3026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance:
uuid: "e3049ca6595d424f88b45463dc85629f"
format_stamp: "Formatted at 2025-10-26 15:13:49 on dist-test-slave-8pgz"
server_key: "495c9a2ba251c7b91520bce2fce83eb3"
server_key_iv: "20be0cfce28013438f9694b1be0d51ba"
server_key_version: "encryptionkey@0"
I20251026 15:13:49.248636 3026 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:13:49.249488 3040 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:49.249696 3026 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:49.249775 3026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/wal
uuid: "e3049ca6595d424f88b45463dc85629f"
format_stamp: "Formatted at 2025-10-26 15:13:49 on dist-test-slave-8pgz"
server_key: "495c9a2ba251c7b91520bce2fce83eb3"
server_key_iv: "20be0cfce28013438f9694b1be0d51ba"
server_key_version: "encryptionkey@0"
I20251026 15:13:49.249836 3026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:49.258801 3026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:49.259162 3026 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:49.259301 3026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:49.259526 3026 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 15:13:49.259920 3026 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 15:13:49.259959 3026 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:49.259989 3026 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 15:13:49.260007 3026 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:49.267189 3026 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.67:45621
I20251026 15:13:49.267249 3153 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.67:45621 every 8 connection(s)
I20251026 15:13:49.267549 3026 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/data/info.pb
I20251026 15:13:49.270483 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 3026
I20251026 15:13:49.270597 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/wal/instance
I20251026 15:13:49.270777 15185 external_mini_cluster.cc:1468] Setting key 6376b001887bed933f0a96c8d6c21499
I20251026 15:13:49.272565 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.268294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.67:60365 (local address 127.14.212.126:46785)
1026 15:13:49.268345 (+ 51us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:49.268347 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:49.268556 (+ 209us) server_negotiation.cc:408] Connection header received
1026 15:13:49.269143 (+ 587us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:49.269144 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:49.269158 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:49.269263 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:49.270021 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.270773 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.271470 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.271643 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.272285 (+ 642us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:49.272288 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:49.272289 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:49.272296 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:49.272310 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:49.272313 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:49.272362 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:49.272455 (+ 93us) server_negotiation.cc:300] Negotiation successful
1026 15:13:49.272491 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20251026 15:13:49.272718 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.268259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:46785 (local address 127.14.212.67:60365)
1026 15:13:49.268455 (+ 196us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:49.268483 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:49.269052 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:49.269336 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:49.269341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:49.269352 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:13:49.269750 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.269754 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.270896 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.270897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:49.271382 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.271385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.271530 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.272159 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1026 15:13:49.272160 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:13:49.272181 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1026 15:13:49.272200 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:13:49.272369 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:13:49.272372 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:13:49.272376 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:49.272453 (+ 77us) client_negotiation.cc:240] Negotiation successful
1026 15:13:49.272533 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20251026 15:13:49.273154 3154 heartbeater.cc:344] Connected to a master server at 127.14.212.126:46785
I20251026 15:13:49.273231 3154 heartbeater.cc:461] Registering TS with master...
I20251026 15:13:49.273397 3154 heartbeater.cc:507] Master 127.14.212.126:46785 requested a full tablet report, sending...
I20251026 15:13:49.273694 2704 ts_manager.cc:194] Registered new tserver with Master: e3049ca6595d424f88b45463dc85629f (127.14.212.67:45621)
I20251026 15:13:49.274070 2704 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.14.212.67:60365
I20251026 15:13:49.282845 15185 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251026 15:13:49.286952 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.284300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.14.212.126:46785)
1026 15:13:49.284365 (+ 65us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:49.284368 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:49.284403 (+ 35us) server_negotiation.cc:408] Connection header received
1026 15:13:49.284495 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:49.284495 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:49.284505 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:49.284551 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:49.284906 (+ 355us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.285420 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.286009 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.286214 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.286709 (+ 495us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:49.286713 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:49.286713 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:49.286719 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:49.286731 (+ 12us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:49.286735 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:49.286789 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:49.286847 (+ 58us) server_negotiation.cc:300] Negotiation successful
1026 15:13:49.286873 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251026 15:13:49.288780 2704 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 15:13:49.290005 2704 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 15:13:49.295598 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.292052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.66:36173 (local address 127.0.0.1:47744)
1026 15:13:49.292776 (+ 724us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:49.292784 (+ 8us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:49.292830 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:49.293097 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:49.293098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:49.293099 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:49.293334 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.293336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.294558 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.294559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:49.295316 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.295318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.295385 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.295421 (+ 36us) client_negotiation.cc:769] Sending connection context
1026 15:13:49.295471 (+ 50us) client_negotiation.cc:240] Negotiation successful
1026 15:13:49.295516 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":115,"threads_started":1}
I20251026 15:13:49.296140 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.292296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.14.212.66:36173)
1026 15:13:49.292518 (+ 222us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:49.292526 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:49.292896 (+ 370us) server_negotiation.cc:408] Connection header received
1026 15:13:49.292937 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:49.292939 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:49.292956 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:49.293014 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:49.293403 (+ 389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.294486 (+ 1083us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.295389 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.295878 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.295972 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:49.296008 (+ 36us) server_negotiation.cc:300] Negotiation successful
1026 15:13:49.296050 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20251026 15:13:49.296197 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.292050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:45621 (local address 127.0.0.1:38264)
1026 15:13:49.292410 (+ 360us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:49.292441 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:49.292524 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:49.292825 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:49.292828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:49.292830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:49.293181 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.293182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.294990 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.294992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:49.295981 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.295984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.296045 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.296049 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:49.296083 (+ 34us) client_negotiation.cc:240] Negotiation successful
1026 15:13:49.296121 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":72,"threads_started":1}
I20251026 15:13:49.296654 2957 tablet_service.cc:1505] Processing CreateTablet for tablet 271736f54a854ca8ad0b7915956e7527 (DEFAULT_TABLE table=test-table [id=3f20bbac421c413fbafea5cf9f1aa8a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:49.296841 3162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.292052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:36637 (local address 127.0.0.1:33684)
1026 15:13:49.292612 (+ 560us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:49.292622 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:49.292696 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:49.292987 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:49.292988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:49.292990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:49.293164 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.293166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.294999 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.295000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:49.295981 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.295984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.296674 (+ 690us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.296677 (+ 3us) client_negotiation.cc:769] Sending connection context
1026 15:13:49.296743 (+ 66us) client_negotiation.cc:240] Negotiation successful
1026 15:13:49.296782 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":103,"threads_started":1}
I20251026 15:13:49.296914 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.292245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.14.212.65:36637)
1026 15:13:49.292416 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:49.292424 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:49.292733 (+ 309us) server_negotiation.cc:408] Connection header received
1026 15:13:49.292774 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:49.292777 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:49.292798 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:49.292872 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:49.293265 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.294884 (+ 1619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.296079 (+ 1195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.296566 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.296644 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:49.296783 (+ 139us) server_negotiation.cc:300] Negotiation successful
1026 15:13:49.296828 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20251026 15:13:49.296957 2957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 271736f54a854ca8ad0b7915956e7527. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:49.297058 3167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.292287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38264 (local address 127.14.212.67:45621)
1026 15:13:49.292611 (+ 324us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:49.292619 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:49.292632 (+ 13us) server_negotiation.cc:408] Connection header received
1026 15:13:49.292709 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:49.292712 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:49.292734 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:49.292809 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:49.293266 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.294881 (+ 1615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.296365 (+ 1484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.296839 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.296917 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:49.296948 (+ 31us) server_negotiation.cc:300] Negotiation successful
1026 15:13:49.296983 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":102,"threads_started":1}
I20251026 15:13:49.297542 2814 tablet_service.cc:1505] Processing CreateTablet for tablet 271736f54a854ca8ad0b7915956e7527 (DEFAULT_TABLE table=test-table [id=3f20bbac421c413fbafea5cf9f1aa8a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:49.297542 3088 tablet_service.cc:1505] Processing CreateTablet for tablet 271736f54a854ca8ad0b7915956e7527 (DEFAULT_TABLE table=test-table [id=3f20bbac421c413fbafea5cf9f1aa8a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 15:13:49.297840 3088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 271736f54a854ca8ad0b7915956e7527. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:49.297840 2814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 271736f54a854ca8ad0b7915956e7527. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:49.299400 3168 tablet_bootstrap.cc:492] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273: Bootstrap starting.
I20251026 15:13:49.300092 3168 tablet_bootstrap.cc:654] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:49.300361 3168 log.cc:826] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:49.300573 3170 tablet_bootstrap.cc:492] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f: Bootstrap starting.
I20251026 15:13:49.301090 3168 tablet_bootstrap.cc:492] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273: No bootstrap required, opened a new log
I20251026 15:13:49.301153 3168 ts_tablet_manager.cc:1403] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:13:49.301160 3170 tablet_bootstrap.cc:654] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:49.301398 3170 log.cc:826] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:49.301698 3169 tablet_bootstrap.cc:492] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f: Bootstrap starting.
I20251026 15:13:49.302114 3170 tablet_bootstrap.cc:492] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f: No bootstrap required, opened a new log
I20251026 15:13:49.302181 3170 ts_tablet_manager.cc:1403] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 15:13:49.302312 3169 tablet_bootstrap.cc:654] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:49.302343 3168 raft_consensus.cc:359] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65970c705294f9087444d359228e13f" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 36637 } } peers { permanent_uuid: "e3049ca6595d424f88b45463dc85629f" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45621 } } peers { permanent_uuid: "7fa088e064c24a328d23a0849f618273" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36173 } }
I20251026 15:13:49.302518 3168 raft_consensus.cc:385] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:49.302557 3168 raft_consensus.cc:740] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa088e064c24a328d23a0849f618273, State: Initialized, Role: FOLLOWER
I20251026 15:13:49.302690 3169 log.cc:826] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:49.302649 3168 consensus_queue.cc:260] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65970c705294f9087444d359228e13f" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 36637 } } peers { permanent_uuid: "e3049ca6595d424f88b45463dc85629f" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45621 } } peers { permanent_uuid: "7fa088e064c24a328d23a0849f618273" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36173 } }
I20251026 15:13:49.302892 3168 ts_tablet_manager.cc:1434] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:13:49.303016 3023 heartbeater.cc:499] Master 127.14.212.126:46785 was elected leader, sending a full tablet report...
I20251026 15:13:49.303673 3170 raft_consensus.cc:359] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65970c705294f9087444d359228e13f" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 36637 } } peers { permanent_uuid: "e3049ca6595d424f88b45463dc85629f" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45621 } } peers { permanent_uuid: "7fa088e064c24a328d23a0849f618273" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36173 } }
I20251026 15:13:49.303817 3170 raft_consensus.cc:385] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:49.303843 3170 raft_consensus.cc:740] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c65970c705294f9087444d359228e13f, State: Initialized, Role: FOLLOWER
I20251026 15:13:49.303930 3170 consensus_queue.cc:260] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65970c705294f9087444d359228e13f" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 36637 } } peers { permanent_uuid: "e3049ca6595d424f88b45463dc85629f" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45621 } } peers { permanent_uuid: "7fa088e064c24a328d23a0849f618273" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36173 } }
I20251026 15:13:49.304149 3170 ts_tablet_manager.cc:1434] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 15:13:49.304231 2892 heartbeater.cc:499] Master 127.14.212.126:46785 was elected leader, sending a full tablet report...
I20251026 15:13:49.304867 3169 tablet_bootstrap.cc:492] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f: No bootstrap required, opened a new log
I20251026 15:13:49.304940 3169 ts_tablet_manager.cc:1403] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20251026 15:13:49.306247 3169 raft_consensus.cc:359] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65970c705294f9087444d359228e13f" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 36637 } } peers { permanent_uuid: "e3049ca6595d424f88b45463dc85629f" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45621 } } peers { permanent_uuid: "7fa088e064c24a328d23a0849f618273" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36173 } }
I20251026 15:13:49.306367 3169 raft_consensus.cc:385] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:49.306418 3169 raft_consensus.cc:740] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3049ca6595d424f88b45463dc85629f, State: Initialized, Role: FOLLOWER
I20251026 15:13:49.306510 3169 consensus_queue.cc:260] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65970c705294f9087444d359228e13f" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 36637 } } peers { permanent_uuid: "e3049ca6595d424f88b45463dc85629f" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45621 } } peers { permanent_uuid: "7fa088e064c24a328d23a0849f618273" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36173 } }
I20251026 15:13:49.306948 3169 ts_tablet_manager.cc:1434] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251026 15:13:49.307015 3154 heartbeater.cc:499] Master 127.14.212.126:46785 was elected leader, sending a full tablet report...
I20251026 15:13:49.350080 3173 raft_consensus.cc:493] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 15:13:49.350250 3173 raft_consensus.cc:515] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65970c705294f9087444d359228e13f" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 36637 } } peers { permanent_uuid: "e3049ca6595d424f88b45463dc85629f" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45621 } } peers { permanent_uuid: "7fa088e064c24a328d23a0849f618273" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36173 } }
I20251026 15:13:49.350721 3173 leader_election.cc:290] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c65970c705294f9087444d359228e13f (127.14.212.65:36637), e3049ca6595d424f88b45463dc85629f (127.14.212.67:45621)
I20251026 15:13:49.353605 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.350972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.67:45621 (local address 127.14.212.66:52137)
1026 15:13:49.351079 (+ 107us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:49.351093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:49.351160 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:49.351384 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:49.351385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:49.351387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:49.351592 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.351594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.352428 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.352429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:49.353368 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.353370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.353466 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.353471 (+ 5us) client_negotiation.cc:769] Sending connection context
1026 15:13:49.353493 (+ 22us) client_negotiation.cc:240] Negotiation successful
1026 15:13:49.353526 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20251026 15:13:49.354331 3177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.350971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:36637 (local address 127.14.212.66:58913)
1026 15:13:49.351237 (+ 266us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:49.351248 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:13:49.351330 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:13:49.351485 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:13:49.351485 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:13:49.351486 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1026 15:13:49.351944 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.351946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.352780 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.352781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:13:49.353673 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:13:49.353676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.354187 (+ 511us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.354191 (+ 4us) client_negotiation.cc:769] Sending connection context
1026 15:13:49.354239 (+ 48us) client_negotiation.cc:240] Negotiation successful
1026 15:13:49.354273 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":14,"thread_start_us":170,"threads_started":1}
I20251026 15:13:49.354421 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.351040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:58913 (local address 127.14.212.65:36637)
1026 15:13:49.351110 (+ 70us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:49.351112 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:49.351362 (+ 250us) server_negotiation.cc:408] Connection header received
1026 15:13:49.351394 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:49.351395 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:49.351406 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:49.351482 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:49.352050 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.352668 (+ 618us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.353777 (+ 1109us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.354146 (+ 369us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.354154 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:49.354294 (+ 140us) server_negotiation.cc:300] Negotiation successful
1026 15:13:49.354340 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251026 15:13:49.354444 3167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.351048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.212.66:52137 (local address 127.14.212.67:45621)
1026 15:13:49.351093 (+ 45us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:49.351096 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:49.351160 (+ 64us) server_negotiation.cc:408] Connection header received
1026 15:13:49.351250 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:49.351252 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:49.351265 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:49.351336 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1026 15:13:49.351681 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.352344 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.353539 (+ 1195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.354250 (+ 711us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.354260 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:49.354288 (+ 28us) server_negotiation.cc:300] Negotiation successful
1026 15:13:49.354325 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251026 15:13:49.354714 3108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "271736f54a854ca8ad0b7915956e7527" candidate_uuid: "7fa088e064c24a328d23a0849f618273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3049ca6595d424f88b45463dc85629f" is_pre_election: true
I20251026 15:13:49.354786 2831 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "271736f54a854ca8ad0b7915956e7527" candidate_uuid: "7fa088e064c24a328d23a0849f618273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c65970c705294f9087444d359228e13f" is_pre_election: true
I20251026 15:13:49.354895 3108 raft_consensus.cc:2468] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa088e064c24a328d23a0849f618273 in term 0.
I20251026 15:13:49.354902 2831 raft_consensus.cc:2468] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa088e064c24a328d23a0849f618273 in term 0.
I20251026 15:13:49.355151 2913 leader_election.cc:304] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa088e064c24a328d23a0849f618273, c65970c705294f9087444d359228e13f; no voters:
I20251026 15:13:49.355346 3173 raft_consensus.cc:2804] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 15:13:49.355419 3173 raft_consensus.cc:493] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 15:13:49.355438 3173 raft_consensus.cc:3060] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:49.356137 3173 raft_consensus.cc:515] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65970c705294f9087444d359228e13f" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 36637 } } peers { permanent_uuid: "e3049ca6595d424f88b45463dc85629f" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45621 } } peers { permanent_uuid: "7fa088e064c24a328d23a0849f618273" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36173 } }
I20251026 15:13:49.356266 3173 leader_election.cc:290] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [CANDIDATE]: Term 1 election: Requested vote from peers c65970c705294f9087444d359228e13f (127.14.212.65:36637), e3049ca6595d424f88b45463dc85629f (127.14.212.67:45621)
I20251026 15:13:49.356457 2831 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "271736f54a854ca8ad0b7915956e7527" candidate_uuid: "7fa088e064c24a328d23a0849f618273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c65970c705294f9087444d359228e13f"
I20251026 15:13:49.356475 3108 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "271736f54a854ca8ad0b7915956e7527" candidate_uuid: "7fa088e064c24a328d23a0849f618273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3049ca6595d424f88b45463dc85629f"
I20251026 15:13:49.356546 2831 raft_consensus.cc:3060] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:49.356545 3108 raft_consensus.cc:3060] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:49.357868 3108 raft_consensus.cc:2468] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa088e064c24a328d23a0849f618273 in term 1.
I20251026 15:13:49.357868 2831 raft_consensus.cc:2468] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa088e064c24a328d23a0849f618273 in term 1.
I20251026 15:13:49.358072 2913 leader_election.cc:304] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa088e064c24a328d23a0849f618273, c65970c705294f9087444d359228e13f; no voters:
I20251026 15:13:49.358165 3173 raft_consensus.cc:2804] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:49.358311 3173 raft_consensus.cc:697] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa088e064c24a328d23a0849f618273, State: Running, Role: LEADER
I20251026 15:13:49.358425 3173 consensus_queue.cc:237] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65970c705294f9087444d359228e13f" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 36637 } } peers { permanent_uuid: "e3049ca6595d424f88b45463dc85629f" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45621 } } peers { permanent_uuid: "7fa088e064c24a328d23a0849f618273" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36173 } }
I20251026 15:13:49.359318 2703 catalog_manager.cc:5649] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa088e064c24a328d23a0849f618273 (127.14.212.66). New cstate: current_term: 1 leader_uuid: "7fa088e064c24a328d23a0849f618273" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65970c705294f9087444d359228e13f" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 36637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3049ca6595d424f88b45463dc85629f" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fa088e064c24a328d23a0849f618273" member_type: VOTER last_known_addr { host: "127.14.212.66" port: 36173 } health_report { overall_health: HEALTHY } } }
I20251026 15:13:49.371994 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.369963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.14.212.66:36173)
1026 15:13:49.370021 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:49.370023 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:49.370030 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:13:49.370069 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:49.370069 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:49.370085 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:49.370143 (+ 58us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:13:49.370379 (+ 236us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.370969 (+ 590us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:49.371541 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:49.371671 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:49.371708 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:13:49.371800 (+ 92us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:13:49.371864 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:49.371904 (+ 40us) server_negotiation.cc:300] Negotiation successful
1026 15:13:49.371926 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251026 15:13:49.373631 3108 raft_consensus.cc:1275] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f [term 1 FOLLOWER]: Refusing update from remote peer 7fa088e064c24a328d23a0849f618273: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:49.373651 2831 raft_consensus.cc:1275] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f [term 1 FOLLOWER]: Refusing update from remote peer 7fa088e064c24a328d23a0849f618273: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:13:49.373924 3173 consensus_queue.cc:1048] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3049ca6595d424f88b45463dc85629f" member_type: VOTER last_known_addr { host: "127.14.212.67" port: 45621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:49.374063 3178 consensus_queue.cc:1048] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c65970c705294f9087444d359228e13f" member_type: VOTER last_known_addr { host: "127.14.212.65" port: 36637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:13:49.376554 3181 mvcc.cc:204] Tried to move back new op lower bound from 7215069713912020992 to 7215069713852686336. Current Snapshot: MvccSnapshot[applied={T|T < 7215069713912020992}]
I20251026 15:13:49.376787 3183 mvcc.cc:204] Tried to move back new op lower bound from 7215069713912020992 to 7215069713852686336. Current Snapshot: MvccSnapshot[applied={T|T < 7215069713912020992}]
I20251026 15:13:49.378633 2703 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60596:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:49.378734 2703 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60596:
table { table_name: "test-table" } modify_external_catalogs: true
I20251026 15:13:49.379482 2703 catalog_manager.cc:5936] T 00000000000000000000000000000000 P d657eeee047e4e20ac9278a12c5dfca5: Sending DeleteTablet for 3 replicas of tablet 271736f54a854ca8ad0b7915956e7527
I20251026 15:13:49.379876 2814 tablet_service.cc:1552] Processing DeleteTablet for tablet 271736f54a854ca8ad0b7915956e7527 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:49 UTC) from {username='slave'} at 127.0.0.1:33684
I20251026 15:13:49.379963 3088 tablet_service.cc:1552] Processing DeleteTablet for tablet 271736f54a854ca8ad0b7915956e7527 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:49 UTC) from {username='slave'} at 127.0.0.1:38264
I20251026 15:13:49.380128 3191 tablet_replica.cc:333] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f: stopping tablet replica
I20251026 15:13:49.380220 3191 raft_consensus.cc:2243] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:49.380239 2957 tablet_service.cc:1552] Processing DeleteTablet for tablet 271736f54a854ca8ad0b7915956e7527 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 15:13:49 UTC) from {username='slave'} at 127.0.0.1:47744
I20251026 15:13:49.380326 3191 raft_consensus.cc:2272] T 271736f54a854ca8ad0b7915956e7527 P c65970c705294f9087444d359228e13f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:49.380496 3192 tablet_replica.cc:333] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f: stopping tablet replica
I20251026 15:13:49.380577 3192 raft_consensus.cc:2243] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 15:13:49.380673 3192 raft_consensus.cc:2272] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:49.380705 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 2764
I20251026 15:13:49.380820 3193 tablet_replica.cc:333] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273: stopping tablet replica
I20251026 15:13:49.380906 3193 raft_consensus.cc:2243] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [term 1 LEADER]: Raft consensus shutting down.
I20251026 15:13:49.380985 3193 raft_consensus.cc:2272] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 15:13:49.381738 3193 ts_tablet_manager.cc:1916] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:49.382128 3192 ts_tablet_manager.cc:1916] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 15:13:49.383509 3193 ts_tablet_manager.cc:1929] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:49.383582 3193 log.cc:1199] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-1/wal/wals/271736f54a854ca8ad0b7915956e7527
I20251026 15:13:49.383893 3193 ts_tablet_manager.cc:1950] T 271736f54a854ca8ad0b7915956e7527 P 7fa088e064c24a328d23a0849f618273: Deleting consensus metadata
I20251026 15:13:49.384306 2689 catalog_manager.cc:4985] TS 7fa088e064c24a328d23a0849f618273 (127.14.212.66:36173): tablet 271736f54a854ca8ad0b7915956e7527 (table test-table [id=3f20bbac421c413fbafea5cf9f1aa8a0]) successfully deleted
I20251026 15:13:49.384349 3192 ts_tablet_manager.cc:1929] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251026 15:13:49.384527 3192 log.cc:1199] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f: Deleting WAL directory at /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761491252308956-15185-0/minicluster-data/ts-2/wal/wals/271736f54a854ca8ad0b7915956e7527
I20251026 15:13:49.384840 3192 ts_tablet_manager.cc:1950] T 271736f54a854ca8ad0b7915956e7527 P e3049ca6595d424f88b45463dc85629f: Deleting consensus metadata
I20251026 15:13:49.385295 2688 catalog_manager.cc:4985] TS e3049ca6595d424f88b45463dc85629f (127.14.212.67:45621): tablet 271736f54a854ca8ad0b7915956e7527 (table test-table [id=3f20bbac421c413fbafea5cf9f1aa8a0]) successfully deleted
W20251026 15:13:49.387519 2691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.212.65:36637 (error 108)
I20251026 15:13:49.387775 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 2895
I20251026 15:13:49.388042 3162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:49.387920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.65:36637 (local address 127.0.0.1:33700)
1026 15:13:49.387968 (+ 48us) negotiation.cc:107] Waiting for socket to connect
1026 15:13:49.388002 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.65:36637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20251026 15:13:49.389128 2691 catalog_manager.cc:4712] TS c65970c705294f9087444d359228e13f (127.14.212.65:36637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 271736f54a854ca8ad0b7915956e7527: Network error: Client connection negotiation failed: client connection to 127.14.212.65:36637: connect: Connection refused (error 111)
I20251026 15:13:49.392920 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 3026
I20251026 15:13:49.397944 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 2673
2025-10-26T15:13:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (664 ms)
[----------] 6 tests from EncryptionPolicyTest (11386 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:13:49 dist-test-slave-8pgz krb5kdc[3197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:13:49 dist-test-slave-8pgz krb5kdc[3197](info): set up 2 sockets
Oct 26 15:13:49 dist-test-slave-8pgz krb5kdc[3197](info): commencing operation
krb5kdc: starting...
W20251026 15:13:51.387979 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.970s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:13:51 dist-test-slave-8pgz krb5kdc[3197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:13:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:13:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:51.529085 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:51.607764 3213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:51.607962 3213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:51.607997 3213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:51.609431 3213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:13:51.609478 3213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:51.609498 3213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:51.609519 3213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:13:51.609541 3213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:13:51.611222 3213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:51.611460 3213 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:51.611632 3213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:51.614210 3219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:51.614218 3218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:51.614444 3213 server_base.cc:1047] running on GCE node
W20251026 15:13:51.614219 3221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:51.614696 3213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:51.614902 3213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:51.616026 3213 hybrid_clock.cc:648] HybridClock initialized: now 1761491631616014 us; error 29 us; skew 500 ppm
Oct 26 15:13:51 dist-test-slave-8pgz krb5kdc[3197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:51.618511 3213 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251026 15:13:51.619057 3213 webserver.cc:492] Webserver started at http://127.0.0.1:46657/ using document root <none> and password file <none>
I20251026 15:13:51.619249 3213 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:51.619289 3213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:51.619405 3213 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:51.620334 3213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "6c06c40116864a51b48c989d33aaef76"
format_stamp: "Formatted at 2025-10-26 15:13:51 on dist-test-slave-8pgz"
server_key: "8b4698c1181e4225813624b335b4593f"
server_key_iv: "a85223a00732d181cd0c88dd1bb3b7c2"
server_key_version: "encryptionkey@0"
I20251026 15:13:51.620631 3213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "6c06c40116864a51b48c989d33aaef76"
format_stamp: "Formatted at 2025-10-26 15:13:51 on dist-test-slave-8pgz"
server_key: "8b4698c1181e4225813624b335b4593f"
server_key_iv: "a85223a00732d181cd0c88dd1bb3b7c2"
server_key_version: "encryptionkey@0"
I20251026 15:13:51.621832 3213 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:13:51.622586 3228 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:51.622763 3213 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:51.622839 3213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "6c06c40116864a51b48c989d33aaef76"
format_stamp: "Formatted at 2025-10-26 15:13:51 on dist-test-slave-8pgz"
server_key: "8b4698c1181e4225813624b335b4593f"
server_key_iv: "a85223a00732d181cd0c88dd1bb3b7c2"
server_key_version: "encryptionkey@0"
I20251026 15:13:51.622900 3213 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:51.644496 3213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:51.651566 3213 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:51.651729 3213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:51.656139 3213 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35977
I20251026 15:13:51.656174 3280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:35977 every 8 connection(s)
I20251026 15:13:51.656505 3213 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:13:51.657088 3281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:51.659530 3281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76: Bootstrap starting.
I20251026 15:13:51.660117 3281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:51.660365 3281 log.cc:826] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:51.661064 3281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76: No bootstrap required, opened a new log
I20251026 15:13:51.662245 3281 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c06c40116864a51b48c989d33aaef76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35977 } }
I20251026 15:13:51.662364 3281 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:51.662384 3281 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c06c40116864a51b48c989d33aaef76, State: Initialized, Role: FOLLOWER
I20251026 15:13:51.662535 3281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c06c40116864a51b48c989d33aaef76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35977 } }
I20251026 15:13:51.662596 3281 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:13:51.662631 3281 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:13:51.662676 3281 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:51.663208 3281 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c06c40116864a51b48c989d33aaef76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35977 } }
I20251026 15:13:51.663307 3281 leader_election.cc:304] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c06c40116864a51b48c989d33aaef76; no voters:
I20251026 15:13:51.663455 3281 leader_election.cc:290] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:13:51.663542 3284 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:51.663688 3284 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [term 1 LEADER]: Becoming Leader. State: Replica: 6c06c40116864a51b48c989d33aaef76, State: Running, Role: LEADER
I20251026 15:13:51.663694 3281 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:13:51.663735 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 3213
I20251026 15:13:51.663806 3284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c06c40116864a51b48c989d33aaef76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35977 } }
I20251026 15:13:51.663846 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:13:51.664045 15185 external_mini_cluster.cc:1468] Setting key a16cb2eb3234680fab1c0e991f9e7315
I20251026 15:13:51.664184 3284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c06c40116864a51b48c989d33aaef76. Latest consensus state: current_term: 1 leader_uuid: "6c06c40116864a51b48c989d33aaef76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c06c40116864a51b48c989d33aaef76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35977 } } }
I20251026 15:13:51.664271 3284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:51.664225 3285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c06c40116864a51b48c989d33aaef76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c06c40116864a51b48c989d33aaef76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35977 } } }
I20251026 15:13:51.664424 3285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:51.665411 3290 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:13:51.666028 3290 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:13:51.667294 3290 catalog_manager.cc:1357] Generated new cluster ID: 7fdc9aa4c7454c7b800e7c95bd419841
I20251026 15:13:51.667346 3290 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 26 15:13:51 dist-test-slave-8pgz krb5kdc[3197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20251026 15:13:51.674264 3290 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:13:51.675031 3290 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:13:51.675221 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:51.664700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54201 (local address 127.0.0.1:35977)
1026 15:13:51.664850 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:51.664872 (+ 22us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:51.664900 (+ 28us) server_negotiation.cc:408] Connection header received
1026 15:13:51.665029 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:51.665035 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:51.665098 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:51.665282 (+ 184us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:51.666244 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:51.667577 (+ 1333us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:51.668334 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:51.668564 (+ 230us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:51.671110 (+ 2546us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:51.671116 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:51.671116 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:51.671140 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:51.673302 (+ 2162us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:51.673750 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:51.673753 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:51.673754 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:51.673819 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:51.674023 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:51.674024 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:51.674024 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:51.674296 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:51.674836 (+ 540us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:51.674940 (+ 104us) server_negotiation.cc:300] Negotiation successful
1026 15:13:51.675050 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20251026 15:13:51.676013 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:13:51.681555 3290 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 6c06c40116864a51b48c989d33aaef76: Generated new TSK 0
I20251026 15:13:51.681742 3290 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:13:51.685334 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:51.677470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40611 (local address 127.0.0.1:35977)
1026 15:13:51.677501 (+ 31us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:51.677503 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:51.678607 (+ 1104us) server_negotiation.cc:408] Connection header received
1026 15:13:51.678641 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:51.678642 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:51.678654 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:51.678719 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:51.679554 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:51.680320 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:51.681114 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:51.681690 (+ 576us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:51.682541 (+ 851us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:51.682545 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:51.682545 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:51.682551 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:51.684146 (+ 1595us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:51.684595 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:51.684596 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:51.684596 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:51.684621 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:51.684794 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:51.684795 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:51.684795 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:51.684902 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:51.685006 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:51.685148 (+ 142us) server_negotiation.cc:300] Negotiation successful
1026 15:13:51.685234 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20251026 15:13:51.687526 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 3213
W20251026 15:13:51.691867 3195 connection.cc:537] client connection to 127.0.0.1:35977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35977): Connection reset by peer (error 104)
2025-10-26T15:13:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:13:51 dist-test-slave-8pgz krb5kdc[3312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:13:51 dist-test-slave-8pgz krb5kdc[3312](info): set up 2 sockets
Oct 26 15:13:51 dist-test-slave-8pgz krb5kdc[3312](info): commencing operation
krb5kdc: starting...
W20251026 15:13:53.675992 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.971s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:13:53 dist-test-slave-8pgz krb5kdc[3312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:13:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:13:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:53.817505 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:53.898098 3328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:53.898329 3328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:53.898367 3328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:53.899950 3328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:13:53.900020 3328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:53.900043 3328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:53.900065 3328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:13:53.900082 3328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:13:53.901845 3328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42859
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:53.902139 3328 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:53.902338 3328 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:53.904955 3334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:53.904960 3333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:53.904979 3336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:53.905213 3328 server_base.cc:1047] running on GCE node
I20251026 15:13:53.905436 3328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:53.905678 3328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:53.906824 3328 hybrid_clock.cc:648] HybridClock initialized: now 1761491633906796 us; error 52 us; skew 500 ppm
Oct 26 15:13:53 dist-test-slave-8pgz krb5kdc[3312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:53.909430 3328 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251026 15:13:53.910023 3328 webserver.cc:492] Webserver started at http://127.0.0.1:33145/ using document root <none> and password file <none>
I20251026 15:13:53.910218 3328 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:53.910256 3328 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:53.910367 3328 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:53.911388 3328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "16fd4c5ad4d041c8bb93c7302783f933"
format_stamp: "Formatted at 2025-10-26 15:13:53 on dist-test-slave-8pgz"
server_key: "99b10ae7eeed8305eab6466248cf0d97"
server_key_iv: "e7a78bfe1610ccb44766e8f55255a3b9"
server_key_version: "encryptionkey@0"
I20251026 15:13:53.911701 3328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "16fd4c5ad4d041c8bb93c7302783f933"
format_stamp: "Formatted at 2025-10-26 15:13:53 on dist-test-slave-8pgz"
server_key: "99b10ae7eeed8305eab6466248cf0d97"
server_key_iv: "e7a78bfe1610ccb44766e8f55255a3b9"
server_key_version: "encryptionkey@0"
I20251026 15:13:53.913025 3328 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:13:53.913776 3343 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:53.913972 3328 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:53.914053 3328 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "16fd4c5ad4d041c8bb93c7302783f933"
format_stamp: "Formatted at 2025-10-26 15:13:53 on dist-test-slave-8pgz"
server_key: "99b10ae7eeed8305eab6466248cf0d97"
server_key_iv: "e7a78bfe1610ccb44766e8f55255a3b9"
server_key_version: "encryptionkey@0"
I20251026 15:13:53.914120 3328 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:53.925707 3328 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:53.927641 3328 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:53.927790 3328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:53.931689 3328 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42859
I20251026 15:13:53.931723 3395 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:42859 every 8 connection(s)
I20251026 15:13:53.932049 3328 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:13:53.932339 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 3328
I20251026 15:13:53.932451 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:13:53.932639 15185 external_mini_cluster.cc:1468] Setting key b39b20cdc4c7a92fc09c6c4862e527bd
I20251026 15:13:53.932672 3396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:53.936153 3396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933: Bootstrap starting.
I20251026 15:13:53.937265 3396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:53.937580 3396 log.cc:826] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:53.937721 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:53.933153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43287 (local address 127.0.0.1:42859)
1026 15:13:53.933316 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:53.933325 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:53.933345 (+ 20us) server_negotiation.cc:408] Connection header received
1026 15:13:53.933727 (+ 382us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:53.933732 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:53.933775 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:53.933889 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:53.934448 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:53.935212 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:53.936028 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:53.936278 (+ 250us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:53.936780 (+ 502us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:53.936817 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:53.936818 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:53.937051 (+ 233us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:53.937078 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:53.937089 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:53.937195 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:53.937441 (+ 246us) server_negotiation.cc:300] Negotiation successful
1026 15:13:53.937470 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20251026 15:13:53.938558 3396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933: No bootstrap required, opened a new log
I20251026 15:13:53.938830 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:13:53.939705 3396 raft_consensus.cc:359] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fd4c5ad4d041c8bb93c7302783f933" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42859 } }
I20251026 15:13:53.939829 3396 raft_consensus.cc:385] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:53.939863 3396 raft_consensus.cc:740] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16fd4c5ad4d041c8bb93c7302783f933, State: Initialized, Role: FOLLOWER
I20251026 15:13:53.939990 3396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fd4c5ad4d041c8bb93c7302783f933" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42859 } }
I20251026 15:13:53.940052 3396 raft_consensus.cc:399] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:13:53.940091 3396 raft_consensus.cc:493] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:13:53.940135 3396 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:53.940776 3396 raft_consensus.cc:515] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fd4c5ad4d041c8bb93c7302783f933" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42859 } }
I20251026 15:13:53.940884 3396 leader_election.cc:304] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16fd4c5ad4d041c8bb93c7302783f933; no voters:
I20251026 15:13:53.941032 3396 leader_election.cc:290] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:13:53.941203 3407 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:53.941270 3396 sys_catalog.cc:565] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:13:53.941536 3407 raft_consensus.cc:697] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [term 1 LEADER]: Becoming Leader. State: Replica: 16fd4c5ad4d041c8bb93c7302783f933, State: Running, Role: LEADER
I20251026 15:13:53.941663 3407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fd4c5ad4d041c8bb93c7302783f933" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42859 } }
I20251026 15:13:53.941987 3407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16fd4c5ad4d041c8bb93c7302783f933. Latest consensus state: current_term: 1 leader_uuid: "16fd4c5ad4d041c8bb93c7302783f933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fd4c5ad4d041c8bb93c7302783f933" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42859 } } }
I20251026 15:13:53.942123 3407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:53.942384 3414 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:13:53.942915 3414 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:13:53.943298 3405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16fd4c5ad4d041c8bb93c7302783f933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fd4c5ad4d041c8bb93c7302783f933" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42859 } } }
I20251026 15:13:53.944725 3405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:53.945492 3414 catalog_manager.cc:1357] Generated new cluster ID: 1c484e1e4d20421a97e3a5178aeb4ca5
I20251026 15:13:53.945538 3414 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:13:53.946556 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:53.941239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53933 (local address 127.0.0.1:42859)
1026 15:13:53.941277 (+ 38us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:53.941279 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:53.941391 (+ 112us) server_negotiation.cc:408] Connection header received
1026 15:13:53.941490 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:53.941491 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:53.941501 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:53.941531 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:53.941949 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:53.942963 (+ 1014us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:53.943788 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:53.943977 (+ 189us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:53.944482 (+ 505us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:53.944485 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:53.944485 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:53.944490 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:53.944505 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:53.944509 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:53.944584 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:53.946462 (+ 1878us) server_negotiation.cc:300] Negotiation successful
1026 15:13:53.946476 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251026 15:13:53.961951 3414 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:13:53.962596 3414 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:13:53.966755 3414 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 16fd4c5ad4d041c8bb93c7302783f933: Generated new TSK 0
I20251026 15:13:53.966989 3414 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:13:53.972808 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 3328
W20251026 15:13:53.977396 3310 connection.cc:537] client connection to 127.0.0.1:42859 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42859): Connection reset by peer (error 104)
2025-10-26T15:13:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:13:54 dist-test-slave-8pgz krb5kdc[3427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:13:54 dist-test-slave-8pgz krb5kdc[3427](info): set up 2 sockets
Oct 26 15:13:54 dist-test-slave-8pgz krb5kdc[3427](info): commencing operation
krb5kdc: starting...
W20251026 15:13:55.980216 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.989s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:13:56 dist-test-slave-8pgz krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:13:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:13:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:56.121232 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46621
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:56.204595 3443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:56.204808 3443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:56.204849 3443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:56.206336 3443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:13:56.206416 3443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:56.206441 3443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:56.206462 3443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:13:56.206492 3443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:13:56.208160 3443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46621
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:56.208428 3443 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:56.208616 3443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:56.211128 3449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:56.211134 3448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:56.211145 3451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:56.211387 3443 server_base.cc:1047] running on GCE node
I20251026 15:13:56.211582 3443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:56.211784 3443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:56.212949 3443 hybrid_clock.cc:648] HybridClock initialized: now 1761491636212862 us; error 107 us; skew 500 ppm
Oct 26 15:13:56 dist-test-slave-8pgz krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:56.215466 3443 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251026 15:13:56.215987 3443 webserver.cc:492] Webserver started at http://127.0.0.1:45137/ using document root <none> and password file <none>
I20251026 15:13:56.216178 3443 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:56.216210 3443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:56.216311 3443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:56.217276 3443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "1b9e76258bf9465291f138724a8e87a2"
format_stamp: "Formatted at 2025-10-26 15:13:56 on dist-test-slave-8pgz"
server_key: "90708cf3d3bbf2eb9591de58b4b1638f"
server_key_iv: "ce98df7c49bbac43d91d0f9293f57b1d"
server_key_version: "encryptionkey@0"
I20251026 15:13:56.217561 3443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "1b9e76258bf9465291f138724a8e87a2"
format_stamp: "Formatted at 2025-10-26 15:13:56 on dist-test-slave-8pgz"
server_key: "90708cf3d3bbf2eb9591de58b4b1638f"
server_key_iv: "ce98df7c49bbac43d91d0f9293f57b1d"
server_key_version: "encryptionkey@0"
I20251026 15:13:56.218868 3443 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:13:56.219616 3458 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:56.219784 3443 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:13:56.219859 3443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "1b9e76258bf9465291f138724a8e87a2"
format_stamp: "Formatted at 2025-10-26 15:13:56 on dist-test-slave-8pgz"
server_key: "90708cf3d3bbf2eb9591de58b4b1638f"
server_key_iv: "ce98df7c49bbac43d91d0f9293f57b1d"
server_key_version: "encryptionkey@0"
I20251026 15:13:56.219925 3443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:56.223352 3443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:56.225198 3443 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:56.225296 3443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:56.229192 3443 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46621
I20251026 15:13:56.229223 3510 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:46621 every 8 connection(s)
I20251026 15:13:56.229529 3443 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:13:56.230192 3511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:56.232703 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2: Bootstrap starting.
I20251026 15:13:56.233273 3511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:56.233479 3511 log.cc:826] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:56.234207 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2: No bootstrap required, opened a new log
I20251026 15:13:56.235477 3511 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b9e76258bf9465291f138724a8e87a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46621 } }
I20251026 15:13:56.235607 3511 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:56.235641 3511 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b9e76258bf9465291f138724a8e87a2, State: Initialized, Role: FOLLOWER
I20251026 15:13:56.235767 3511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b9e76258bf9465291f138724a8e87a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46621 } }
I20251026 15:13:56.235812 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 3443
I20251026 15:13:56.235846 3511 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:13:56.235908 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:13:56.235945 3511 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:13:56.235999 3511 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:13:56.236104 15185 external_mini_cluster.cc:1468] Setting key ba5aa6d9f991d8c1bfbbf4729e9b49a5
I20251026 15:13:56.236716 3511 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b9e76258bf9465291f138724a8e87a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46621 } }
I20251026 15:13:56.236842 3511 leader_election.cc:304] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b9e76258bf9465291f138724a8e87a2; no voters:
I20251026 15:13:56.237010 3511 leader_election.cc:290] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:13:56.237059 3516 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:56.237214 3516 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [term 1 LEADER]: Becoming Leader. State: Replica: 1b9e76258bf9465291f138724a8e87a2, State: Running, Role: LEADER
I20251026 15:13:56.237237 3511 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:13:56.237320 3516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b9e76258bf9465291f138724a8e87a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46621 } }
I20251026 15:13:56.237747 3516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b9e76258bf9465291f138724a8e87a2. Latest consensus state: current_term: 1 leader_uuid: "1b9e76258bf9465291f138724a8e87a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b9e76258bf9465291f138724a8e87a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46621 } } }
I20251026 15:13:56.237839 3516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:56.238014 3517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b9e76258bf9465291f138724a8e87a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b9e76258bf9465291f138724a8e87a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46621 } } }
I20251026 15:13:56.238093 3517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2 [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:56.238200 3523 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:13:56.238870 3515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:56.236590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45953 (local address 127.0.0.1:46621)
1026 15:13:56.236758 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:56.236767 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:56.236820 (+ 53us) server_negotiation.cc:408] Connection header received
1026 15:13:56.236945 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:56.236950 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:56.236977 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:56.237105 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:56.238269 (+ 1164us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:56.238273 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:56.238273 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:56.238278 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:56.238302 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:56.238310 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:56.238419 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:56.238663 (+ 244us) server_negotiation.cc:300] Negotiation successful
1026 15:13:56.238702 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20251026 15:13:56.239110 3523 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:13:56.239755 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:13:56.240633 3523 catalog_manager.cc:1357] Generated new cluster ID: 6e322adf48df442da0d4258e50ba6d3a
I20251026 15:13:56.240682 3523 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:13:56.242377 3515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:56.241199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34811 (local address 127.0.0.1:46621)
1026 15:13:56.241243 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:56.241245 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:56.241338 (+ 93us) server_negotiation.cc:408] Connection header received
1026 15:13:56.241410 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:56.241411 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:56.241416 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:56.241471 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:56.242159 (+ 688us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:56.242161 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:56.242163 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:13:56.242167 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:56.242182 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:13:56.242185 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:56.242235 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:56.242297 (+ 62us) server_negotiation.cc:300] Negotiation successful
1026 15:13:56.242312 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20251026 15:13:56.248796 3523 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:13:56.248988 3523 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:13:56.252703 3523 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 1b9e76258bf9465291f138724a8e87a2: Generated new TSK 0
I20251026 15:13:56.252929 3523 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:13:56.254695 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 3443
2025-10-26T15:13:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2281 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:13:56 dist-test-slave-8pgz krb5kdc[3542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:13:56 dist-test-slave-8pgz krb5kdc[3542](info): set up 2 sockets
Oct 26 15:13:56 dist-test-slave-8pgz krb5kdc[3542](info): commencing operation
krb5kdc: starting...
W20251026 15:13:58.288218 15185 mini_kdc.cc:121] Time spent starting KDC: real 2.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:13:58 dist-test-slave-8pgz krb5kdc[3542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:13:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:13:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:13:58.429694 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:13:58.511417 3558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:13:58.511641 3558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:13:58.511687 3558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:13:58.513279 3558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:13:58.513350 3558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:13:58.513371 3558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:13:58.513394 3558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:13:58.513411 3558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:13:58.515290 3558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:13:58.515591 3558 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:13:58.515785 3558 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:13:58.518522 3564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:58.518585 3558 server_base.cc:1047] running on GCE node
W20251026 15:13:58.518504 3563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:13:58.518502 3566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:13:58.518870 3558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:13:58.519110 3558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:13:58.520269 3558 hybrid_clock.cc:648] HybridClock initialized: now 1761491638520251 us; error 35 us; skew 500 ppm
Oct 26 15:13:58 dist-test-slave-8pgz krb5kdc[3542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:13:58.523046 3558 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251026 15:13:58.523658 3558 webserver.cc:492] Webserver started at http://127.0.0.1:41721/ using document root <none> and password file <none>
I20251026 15:13:58.523851 3558 fs_manager.cc:362] Metadata directory not provided
I20251026 15:13:58.523893 3558 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:13:58.524013 3558 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:13:58.524973 3558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "06e25010dd9545a88c614108d7e390cc"
format_stamp: "Formatted at 2025-10-26 15:13:58 on dist-test-slave-8pgz"
server_key: "a5ea9066d78117bb7709c6b00c07554c"
server_key_iv: "79eadfb4d037cef780042de2d0495116"
server_key_version: "encryptionkey@0"
I20251026 15:13:58.525285 3558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "06e25010dd9545a88c614108d7e390cc"
format_stamp: "Formatted at 2025-10-26 15:13:58 on dist-test-slave-8pgz"
server_key: "a5ea9066d78117bb7709c6b00c07554c"
server_key_iv: "79eadfb4d037cef780042de2d0495116"
server_key_version: "encryptionkey@0"
I20251026 15:13:58.526693 3558 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:13:58.527508 3573 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:13:58.527737 3558 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:13:58.527824 3558 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "06e25010dd9545a88c614108d7e390cc"
format_stamp: "Formatted at 2025-10-26 15:13:58 on dist-test-slave-8pgz"
server_key: "a5ea9066d78117bb7709c6b00c07554c"
server_key_iv: "79eadfb4d037cef780042de2d0495116"
server_key_version: "encryptionkey@0"
I20251026 15:13:58.527902 3558 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:13:58.552171 3558 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:13:58.559352 3558 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:13:58.559525 3558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:13:58.563596 3558 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42417
I20251026 15:13:58.563629 3625 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:42417 every 8 connection(s)
I20251026 15:13:58.563966 3558 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:13:58.564669 3626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:13:58.564654 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 3558
I20251026 15:13:58.564791 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:13:58.564980 15185 external_mini_cluster.cc:1468] Setting key 8fc0ba4cfdab3d915d23ec9a262d7f66
I20251026 15:13:58.567600 3626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc: Bootstrap starting.
I20251026 15:13:58.568202 3626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc: Neither blocks nor log segments found. Creating new log.
I20251026 15:13:58.568444 3626 log.cc:826] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc: Log is configured to *not* fsync() on all Append() calls
I20251026 15:13:58.569169 3626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc: No bootstrap required, opened a new log
I20251026 15:13:58.570274 3626 raft_consensus.cc:359] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e25010dd9545a88c614108d7e390cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } }
I20251026 15:13:58.570439 3626 raft_consensus.cc:385] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:13:58.570484 3626 raft_consensus.cc:740] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e25010dd9545a88c614108d7e390cc, State: Initialized, Role: FOLLOWER
I20251026 15:13:58.570600 3626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e25010dd9545a88c614108d7e390cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } }
I20251026 15:13:58.570678 3626 raft_consensus.cc:399] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:13:58.570724 3626 raft_consensus.cc:493] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:13:58.570771 3626 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [term 0 FOLLOWER]: Advancing to term 1
Oct 26 15:13:58 dist-test-slave-8pgz krb5kdc[3542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20251026 15:13:58.571583 3626 raft_consensus.cc:515] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e25010dd9545a88c614108d7e390cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } }
I20251026 15:13:58.571698 3626 leader_election.cc:304] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06e25010dd9545a88c614108d7e390cc; no voters:
I20251026 15:13:58.571857 3626 leader_election.cc:290] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:13:58.571990 3631 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:13:58.572099 3626 sys_catalog.cc:565] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:13:58.572141 3631 raft_consensus.cc:697] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [term 1 LEADER]: Becoming Leader. State: Replica: 06e25010dd9545a88c614108d7e390cc, State: Running, Role: LEADER
I20251026 15:13:58.572250 3631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e25010dd9545a88c614108d7e390cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } }
I20251026 15:13:58.572746 3632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06e25010dd9545a88c614108d7e390cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e25010dd9545a88c614108d7e390cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } } }
I20251026 15:13:58.572825 3632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:58.573243 3640 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:13:58.573745 3640 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:13:58.574574 3633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06e25010dd9545a88c614108d7e390cc. Latest consensus state: current_term: 1 leader_uuid: "06e25010dd9545a88c614108d7e390cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e25010dd9545a88c614108d7e390cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } } }
I20251026 15:13:58.574705 3633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc [sys.catalog]: This master's current role is: LEADER
I20251026 15:13:58.576088 3640 catalog_manager.cc:1357] Generated new cluster ID: f505c05181dc46a8a7d5557a3fbe69ae
I20251026 15:13:58.576135 3640 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:13:58.576154 3629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:58.565545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60651 (local address 127.0.0.1:42417)
1026 15:13:58.565708 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:58.565721 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:58.565787 (+ 66us) server_negotiation.cc:408] Connection header received
1026 15:13:58.566270 (+ 483us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:58.566276 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:58.566323 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:58.566460 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:58.567306 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:58.568066 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:58.568714 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:58.568977 (+ 263us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:58.571442 (+ 2465us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:58.571452 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:58.571452 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:58.571471 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:58.574526 (+ 3055us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:58.574977 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:58.574980 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:58.574981 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:58.575047 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:58.575264 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:58.575265 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:58.575265 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:58.575533 (+ 268us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:58.575714 (+ 181us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:58.575837 (+ 123us) server_negotiation.cc:300] Negotiation successful
1026 15:13:58.575968 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20251026 15:13:58.577070 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:13:58.589670 3629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:13:58.582892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48873 (local address 127.0.0.1:42417)
1026 15:13:58.582935 (+ 43us) server_negotiation.cc:207] Beginning negotiation
1026 15:13:58.582938 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:13:58.583055 (+ 117us) server_negotiation.cc:408] Connection header received
1026 15:13:58.583106 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:13:58.583107 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:13:58.583118 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:13:58.583189 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:13:58.583992 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:58.584754 (+ 762us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:13:58.585463 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:13:58.585656 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:13:58.586912 (+ 1256us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:13:58.586916 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:13:58.586917 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:13:58.586924 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:13:58.588620 (+ 1696us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:58.588984 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:58.588985 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:58.588986 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:58.589010 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:13:58.589152 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:13:58.589153 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:13:58.589153 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:13:58.589262 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:13:58.589354 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1026 15:13:58.589476 (+ 122us) server_negotiation.cc:300] Negotiation successful
1026 15:13:58.589562 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251026 15:13:58.591852 3640 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:13:58.592418 3640 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:13:58.603029 3640 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 06e25010dd9545a88c614108d7e390cc: Generated new TSK 0
I20251026 15:13:58.603266 3640 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:13:58.606053 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 3558
2025-10-26T15:13:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:13:58 dist-test-slave-8pgz krb5kdc[3657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:13:58 dist-test-slave-8pgz krb5kdc[3657](info): set up 2 sockets
Oct 26 15:13:58 dist-test-slave-8pgz krb5kdc[3657](info): commencing operation
krb5kdc: starting...
W20251026 15:14:00.604147 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.980s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:00 dist-test-slave-8pgz krb5kdc[3657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:00.746450 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:37001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:00.827296 3673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:00.827539 3673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:00.827589 3673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:00.829218 3673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:00.829288 3673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:00.829312 3673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:00.829334 3673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:00.829353 3673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:00.831305 3673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:00.831593 3673 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:00.831782 3673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:00.834527 3678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:00.834520 3679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:00.834654 3673 server_base.cc:1047] running on GCE node
W20251026 15:14:00.834524 3681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:00.835001 3673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:00.835238 3673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:00.836396 3673 hybrid_clock.cc:648] HybridClock initialized: now 1761491640836370 us; error 45 us; skew 500 ppm
Oct 26 15:14:00 dist-test-slave-8pgz krb5kdc[3657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:00.839109 3673 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251026 15:14:00.839720 3673 webserver.cc:492] Webserver started at http://127.0.0.1:45757/ using document root <none> and password file <none>
I20251026 15:14:00.839926 3673 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:00.839964 3673 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:00.840070 3673 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:00.841095 3673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "f75aba9c1065452fb90e9964d0ef8b99"
format_stamp: "Formatted at 2025-10-26 15:14:00 on dist-test-slave-8pgz"
server_key: "d4d1076bbce07bd66aca55bd978112ed"
server_key_iv: "0b8044456d39e0b8aae3f44a26e72279"
server_key_version: "encryptionkey@0"
I20251026 15:14:00.841408 3673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "f75aba9c1065452fb90e9964d0ef8b99"
format_stamp: "Formatted at 2025-10-26 15:14:00 on dist-test-slave-8pgz"
server_key: "d4d1076bbce07bd66aca55bd978112ed"
server_key_iv: "0b8044456d39e0b8aae3f44a26e72279"
server_key_version: "encryptionkey@0"
I20251026 15:14:00.842715 3673 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:14:00.843493 3688 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:00.843678 3673 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:14:00.843756 3673 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "f75aba9c1065452fb90e9964d0ef8b99"
format_stamp: "Formatted at 2025-10-26 15:14:00 on dist-test-slave-8pgz"
server_key: "d4d1076bbce07bd66aca55bd978112ed"
server_key_iv: "0b8044456d39e0b8aae3f44a26e72279"
server_key_version: "encryptionkey@0"
I20251026 15:14:00.843823 3673 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:00.854717 3673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:00.856585 3673 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:00.856735 3673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:00.861128 3673 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39161
I20251026 15:14:00.861186 3740 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:39161 every 8 connection(s)
I20251026 15:14:00.861497 3673 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:00.862140 3741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:00.864689 3741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99: Bootstrap starting.
I20251026 15:14:00.865305 3741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:00.865538 3741 log.cc:826] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:00.866288 3741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99: No bootstrap required, opened a new log
I20251026 15:14:00.867585 3741 raft_consensus.cc:359] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75aba9c1065452fb90e9964d0ef8b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39161 } }
I20251026 15:14:00.867710 3741 raft_consensus.cc:385] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:00.867730 3741 raft_consensus.cc:740] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75aba9c1065452fb90e9964d0ef8b99, State: Initialized, Role: FOLLOWER
I20251026 15:14:00.867856 3741 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75aba9c1065452fb90e9964d0ef8b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39161 } }
I20251026 15:14:00.867925 3741 raft_consensus.cc:399] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:00.867947 3741 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:00.867977 3741 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:00.868534 3741 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75aba9c1065452fb90e9964d0ef8b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39161 } }
I20251026 15:14:00.868629 3741 leader_election.cc:304] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f75aba9c1065452fb90e9964d0ef8b99; no voters:
I20251026 15:14:00.868769 3741 leader_election.cc:290] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:00.868937 3744 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:00.869004 3741 sys_catalog.cc:565] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:00.869093 3744 raft_consensus.cc:697] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [term 1 LEADER]: Becoming Leader. State: Replica: f75aba9c1065452fb90e9964d0ef8b99, State: Running, Role: LEADER
I20251026 15:14:00.869252 3744 consensus_queue.cc:237] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75aba9c1065452fb90e9964d0ef8b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39161 } }
I20251026 15:14:00.869647 3746 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75aba9c1065452fb90e9964d0ef8b99. Latest consensus state: current_term: 1 leader_uuid: "f75aba9c1065452fb90e9964d0ef8b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75aba9c1065452fb90e9964d0ef8b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39161 } } }
I20251026 15:14:00.869661 3745 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f75aba9c1065452fb90e9964d0ef8b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75aba9c1065452fb90e9964d0ef8b99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39161 } } }
I20251026 15:14:00.869765 3746 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:00.869771 3745 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:00.870004 3753 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:00.870659 3753 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:00.871286 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 3673
I20251026 15:14:00.871435 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:00.871634 15185 external_mini_cluster.cc:1468] Setting key fefb2d4196ca51fc40e07f97bdab38c7
I20251026 15:14:00.872536 3753 catalog_manager.cc:1357] Generated new cluster ID: 4cf2b9bb2fed418b99568c61e289d729
I20251026 15:14:00.872586 3753 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:00.877250 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:00.872121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59265 (local address 127.0.0.1:39161)
1026 15:14:00.872636 (+ 515us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:00.872649 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:00.872673 (+ 24us) server_negotiation.cc:408] Connection header received
1026 15:14:00.872779 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:00.872784 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:00.872845 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:00.872977 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:00.873452 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:00.874781 (+ 1329us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:00.875642 (+ 861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:00.876322 (+ 680us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:00.876442 (+ 120us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:00.876448 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:00.876448 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:00.876690 (+ 242us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:00.876711 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:00.876719 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:00.876806 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:00.876903 (+ 97us) server_negotiation.cc:300] Negotiation successful
1026 15:14:00.877068 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":85,"threads_started":1}
I20251026 15:14:00.878355 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:00.889360 3753 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:00.889976 3753 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:00.892159 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:00.879928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41045 (local address 127.0.0.1:39161)
1026 15:14:00.879968 (+ 40us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:00.879970 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:00.889183 (+ 9213us) server_negotiation.cc:408] Connection header received
1026 15:14:00.889223 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:00.889224 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:00.889240 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:00.889293 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:00.889650 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:00.890473 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:00.891216 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:00.891802 (+ 586us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:00.891892 (+ 90us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:00.891896 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:00.891896 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:00.891902 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:00.891915 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:00.891919 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:00.891981 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:00.892027 (+ 46us) server_negotiation.cc:300] Negotiation successful
1026 15:14:00.892072 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251026 15:14:00.897053 3753 catalog_manager.cc:6022] T 00000000000000000000000000000000 P f75aba9c1065452fb90e9964d0ef8b99: Generated new TSK 0
I20251026 15:14:00.897282 3753 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:00.901788 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 3673
2025-10-26T15:14:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:00 dist-test-slave-8pgz krb5kdc[3772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:00 dist-test-slave-8pgz krb5kdc[3772](info): set up 2 sockets
Oct 26 15:14:00 dist-test-slave-8pgz krb5kdc[3772](info): commencing operation
krb5kdc: starting...
W20251026 15:14:02.908125 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.987s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:02 dist-test-slave-8pgz krb5kdc[3772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:03.049446 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:03.132961 3788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:03.133162 3788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:03.133188 3788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:03.134701 3788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:03.134754 3788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:03.134769 3788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:03.134783 3788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:03.134794 3788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:03.136472 3788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:03.136706 3788 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:03.136927 3788 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:14:03.139632 3788 server_base.cc:1047] running on GCE node
W20251026 15:14:03.139616 3796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:03.139616 3794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:03.139612 3793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:03.140118 3788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:03.140362 3788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:03.141512 3788 hybrid_clock.cc:648] HybridClock initialized: now 1761491643141490 us; error 35 us; skew 500 ppm
Oct 26 15:14:03 dist-test-slave-8pgz krb5kdc[3772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:03.144109 3788 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251026 15:14:03.144631 3788 webserver.cc:492] Webserver started at http://127.0.0.1:44353/ using document root <none> and password file <none>
I20251026 15:14:03.144797 3788 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:03.144829 3788 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:03.144917 3788 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:03.145843 3788 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "c1e12c7511124935b4990b178cc7db2e"
format_stamp: "Formatted at 2025-10-26 15:14:03 on dist-test-slave-8pgz"
server_key: "daf5995ea93e61749c01fc023f9671bb"
server_key_iv: "740dd9480153681043e5085bbc1127f4"
server_key_version: "encryptionkey@0"
I20251026 15:14:03.146121 3788 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "c1e12c7511124935b4990b178cc7db2e"
format_stamp: "Formatted at 2025-10-26 15:14:03 on dist-test-slave-8pgz"
server_key: "daf5995ea93e61749c01fc023f9671bb"
server_key_iv: "740dd9480153681043e5085bbc1127f4"
server_key_version: "encryptionkey@0"
I20251026 15:14:03.147403 3788 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:14:03.148197 3803 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:03.148408 3788 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:14:03.148488 3788 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "c1e12c7511124935b4990b178cc7db2e"
format_stamp: "Formatted at 2025-10-26 15:14:03 on dist-test-slave-8pgz"
server_key: "daf5995ea93e61749c01fc023f9671bb"
server_key_iv: "740dd9480153681043e5085bbc1127f4"
server_key_version: "encryptionkey@0"
I20251026 15:14:03.148557 3788 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:03.152036 3788 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:03.153847 3788 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:03.153995 3788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:03.157905 3788 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37267
I20251026 15:14:03.157938 3855 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:37267 every 8 connection(s)
I20251026 15:14:03.158257 3788 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:03.158938 3856 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:03.161574 3856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e: Bootstrap starting.
I20251026 15:14:03.162266 3856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:03.162547 3856 log.cc:826] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:03.163295 3856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e: No bootstrap required, opened a new log
I20251026 15:14:03.163869 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 3788
I20251026 15:14:03.163981 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:03.164150 15185 external_mini_cluster.cc:1468] Setting key f0dfb37483144b5eb62bd62815bc5b91
I20251026 15:14:03.165092 3856 raft_consensus.cc:359] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e12c7511124935b4990b178cc7db2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37267 } }
I20251026 15:14:03.165243 3856 raft_consensus.cc:385] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:03.165282 3856 raft_consensus.cc:740] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e12c7511124935b4990b178cc7db2e, State: Initialized, Role: FOLLOWER
I20251026 15:14:03.165419 3856 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e12c7511124935b4990b178cc7db2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37267 } }
I20251026 15:14:03.165506 3856 raft_consensus.cc:399] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:03.165549 3856 raft_consensus.cc:493] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:03.165602 3856 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:03.166415 3856 raft_consensus.cc:515] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e12c7511124935b4990b178cc7db2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37267 } }
I20251026 15:14:03.166482 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:03.164683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58605 (local address 127.0.0.1:37267)
1026 15:14:03.164871 (+ 188us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:03.164881 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:03.164905 (+ 24us) server_negotiation.cc:408] Connection header received
1026 15:14:03.165037 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:03.165042 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:03.165067 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:03.165181 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:03.166064 (+ 883us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:03.166067 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:03.166068 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:03.166072 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:03.166098 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:03.166108 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:03.166200 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:03.166239 (+ 39us) server_negotiation.cc:300] Negotiation successful
1026 15:14:03.166275 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20251026 15:14:03.166535 3856 leader_election.cc:304] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1e12c7511124935b4990b178cc7db2e; no voters:
I20251026 15:14:03.166707 3856 leader_election.cc:290] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:03.166780 3861 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:03.166919 3861 raft_consensus.cc:697] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [term 1 LEADER]: Becoming Leader. State: Replica: c1e12c7511124935b4990b178cc7db2e, State: Running, Role: LEADER
I20251026 15:14:03.166986 3856 sys_catalog.cc:565] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:03.167035 3861 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e12c7511124935b4990b178cc7db2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37267 } }
I20251026 15:14:03.167371 3861 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1e12c7511124935b4990b178cc7db2e. Latest consensus state: current_term: 1 leader_uuid: "c1e12c7511124935b4990b178cc7db2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e12c7511124935b4990b178cc7db2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37267 } } }
I20251026 15:14:03.167490 3861 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:03.167493 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:03.167379 3862 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1e12c7511124935b4990b178cc7db2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e12c7511124935b4990b178cc7db2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37267 } } }
I20251026 15:14:03.167690 3862 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:03.167816 3868 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:03.168533 3868 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:03.170480 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:03.169040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42903 (local address 127.0.0.1:37267)
1026 15:14:03.169076 (+ 36us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:03.169078 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:03.169281 (+ 203us) server_negotiation.cc:408] Connection header received
1026 15:14:03.169380 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:03.169381 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:03.169386 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:03.169455 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:03.170174 (+ 719us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:03.170176 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:03.170176 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:03.170180 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:03.170199 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:03.170203 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:03.170262 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:03.170356 (+ 94us) server_negotiation.cc:300] Negotiation successful
1026 15:14:03.170372 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20251026 15:14:03.171001 3868 catalog_manager.cc:1357] Generated new cluster ID: 6bf71cbc96d54ddb847f55ac65e80657
I20251026 15:14:03.171059 3868 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:03.199000 3868 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:03.199201 3868 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:03.214776 3868 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c1e12c7511124935b4990b178cc7db2e: Generated new TSK 0
I20251026 15:14:03.215027 3868 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:03.224354 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 3788
2025-10-26T15:14:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:03 dist-test-slave-8pgz krb5kdc[3887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:03 dist-test-slave-8pgz krb5kdc[3887](info): set up 2 sockets
Oct 26 15:14:03 dist-test-slave-8pgz krb5kdc[3887](info): commencing operation
krb5kdc: starting...
W20251026 15:14:05.248530 15185 mini_kdc.cc:121] Time spent starting KDC: real 2.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:05 dist-test-slave-8pgz krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:05.402365 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:05.484666 3903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:05.484890 3903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:05.484941 3903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:05.486514 3903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:05.486577 3903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:05.486599 3903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:05.486624 3903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:05.486642 3903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:05.488410 3903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:05.488705 3903 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:05.488898 3903 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:14:05.491818 3903 server_base.cc:1047] running on GCE node
W20251026 15:14:05.491765 3908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:05.491770 3911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:05.491763 3909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:05.492270 3903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:05.492549 3903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:05.493728 3903 hybrid_clock.cc:648] HybridClock initialized: now 1761491645493710 us; error 39 us; skew 500 ppm
Oct 26 15:14:05 dist-test-slave-8pgz krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:05.496480 3903 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251026 15:14:05.497160 3903 webserver.cc:492] Webserver started at http://127.0.0.1:44067/ using document root <none> and password file <none>
I20251026 15:14:05.497378 3903 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:05.497418 3903 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:05.497537 3903 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:05.498731 3903 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "c1c64b96f74e4993a2f8aae397c07af4"
format_stamp: "Formatted at 2025-10-26 15:14:05 on dist-test-slave-8pgz"
server_key: "58d41a809352c5ebe5cb53e41348c640"
server_key_iv: "85da245b8f58183e73b4d811c5c0243b"
server_key_version: "encryptionkey@0"
I20251026 15:14:05.499063 3903 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "c1c64b96f74e4993a2f8aae397c07af4"
format_stamp: "Formatted at 2025-10-26 15:14:05 on dist-test-slave-8pgz"
server_key: "58d41a809352c5ebe5cb53e41348c640"
server_key_iv: "85da245b8f58183e73b4d811c5c0243b"
server_key_version: "encryptionkey@0"
I20251026 15:14:05.500365 3903 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:14:05.501137 3918 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:05.501444 3903 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:14:05.501514 3903 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "c1c64b96f74e4993a2f8aae397c07af4"
format_stamp: "Formatted at 2025-10-26 15:14:05 on dist-test-slave-8pgz"
server_key: "58d41a809352c5ebe5cb53e41348c640"
server_key_iv: "85da245b8f58183e73b4d811c5c0243b"
server_key_version: "encryptionkey@0"
I20251026 15:14:05.501583 3903 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:05.518918 3903 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:05.520838 3903 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:05.521004 3903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:05.525127 3903 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41267
I20251026 15:14:05.525187 3970 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:41267 every 8 connection(s)
I20251026 15:14:05.525496 3903 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:05.526135 3971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:05.527289 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 3903
I20251026 15:14:05.527401 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:05.527563 15185 external_mini_cluster.cc:1468] Setting key 72fe30aab978efc1cfe179ce3962ec6a
I20251026 15:14:05.529232 3971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4: Bootstrap starting.
I20251026 15:14:05.529865 3971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:05.530170 3971 log.cc:826] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:05.530916 3971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4: No bootstrap required, opened a new log
I20251026 15:14:05.532284 3971 raft_consensus.cc:359] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c64b96f74e4993a2f8aae397c07af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41267 } }
I20251026 15:14:05.532415 3971 raft_consensus.cc:385] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:05.532450 3971 raft_consensus.cc:740] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c64b96f74e4993a2f8aae397c07af4, State: Initialized, Role: FOLLOWER
I20251026 15:14:05.532564 3971 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c64b96f74e4993a2f8aae397c07af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41267 } }
I20251026 15:14:05.532636 3971 raft_consensus.cc:399] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:05.532678 3971 raft_consensus.cc:493] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:05.532729 3971 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:05.533489 3971 raft_consensus.cc:515] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c64b96f74e4993a2f8aae397c07af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41267 } }
I20251026 15:14:05.533615 3971 leader_election.cc:304] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1c64b96f74e4993a2f8aae397c07af4; no voters:
Oct 26 15:14:05 dist-test-slave-8pgz krb5kdc[3887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20251026 15:14:05.533782 3971 leader_election.cc:290] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:05.533972 3977 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:05.534050 3971 sys_catalog.cc:565] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:05.534080 3977 raft_consensus.cc:697] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [term 1 LEADER]: Becoming Leader. State: Replica: c1c64b96f74e4993a2f8aae397c07af4, State: Running, Role: LEADER
I20251026 15:14:05.534164 3977 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c64b96f74e4993a2f8aae397c07af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41267 } }
I20251026 15:14:05.534626 3977 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1c64b96f74e4993a2f8aae397c07af4. Latest consensus state: current_term: 1 leader_uuid: "c1c64b96f74e4993a2f8aae397c07af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c64b96f74e4993a2f8aae397c07af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41267 } } }
I20251026 15:14:05.534636 3976 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1c64b96f74e4993a2f8aae397c07af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c64b96f74e4993a2f8aae397c07af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41267 } } }
I20251026 15:14:05.534898 3976 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:05.534816 3977 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4 [sys.catalog]: This master's current role is: LEADER
W20251026 15:14:05.536331 3992 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:14:05.536391 3992 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:14:05.536438 3985 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:05.536598 3985 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:05.538043 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:05.528121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:33273 (local address 127.0.0.1:41267)
1026 15:14:05.528311 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:05.528322 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:05.528346 (+ 24us) server_negotiation.cc:408] Connection header received
1026 15:14:05.528880 (+ 534us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:05.528889 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:05.528955 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:05.529105 (+ 150us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:05.529875 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:05.530966 (+ 1091us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:05.531568 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:05.531824 (+ 256us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:05.534238 (+ 2414us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:05.534247 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:05.534247 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:05.534274 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:05.536514 (+ 2240us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:05.536909 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:05.536911 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:05.536912 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:05.536960 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:05.537145 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:05.537146 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:05.537147 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:05.537344 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:05.537520 (+ 176us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:05.537802 (+ 282us) server_negotiation.cc:300] Negotiation successful
1026 15:14:05.537914 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20251026 15:14:05.538055 3985 catalog_manager.cc:1357] Generated new cluster ID: 4be6910a9be64dbd9ae00ccf01360720
I20251026 15:14:05.538100 3985 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:05.538990 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:05.548694 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:05.542899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:60169 (local address 127.0.0.1:41267)
1026 15:14:05.542966 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:05.542968 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:05.543070 (+ 102us) server_negotiation.cc:408] Connection header received
1026 15:14:05.543144 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:05.543144 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:05.543158 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:05.543208 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:05.543959 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:05.544469 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:05.544984 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:05.545517 (+ 533us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:05.546191 (+ 674us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:05.546195 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:05.546196 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:05.546203 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:05.547643 (+ 1440us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:05.548005 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:05.548006 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:05.548006 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:05.548026 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:05.548192 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:05.548193 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:05.548193 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:05.548280 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:05.548371 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:05.548520 (+ 149us) server_negotiation.cc:300] Negotiation successful
1026 15:14:05.548581 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251026 15:14:05.557482 3985 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:05.558298 3985 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:05.568651 3985 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c1c64b96f74e4993a2f8aae397c07af4: Generated new TSK 0
I20251026 15:14:05.568902 3985 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:05.573542 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 3903
W20251026 15:14:05.577962 3885 connection.cc:537] client connection to 127.0.0.1:41267 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41267): Connection reset by peer (error 104)
2025-10-26T15:14:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:05 dist-test-slave-8pgz krb5kdc[4002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:05 dist-test-slave-8pgz krb5kdc[4002](info): set up 2 sockets
Oct 26 15:14:05 dist-test-slave-8pgz krb5kdc[4002](info): commencing operation
krb5kdc: starting...
W20251026 15:14:07.584288 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.992s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:07 dist-test-slave-8pgz krb5kdc[4002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:07.734480 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:07.817643 4018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:07.817902 4018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:07.817930 4018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:07.819527 4018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:07.819586 4018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:07.819600 4018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:07.819614 4018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:07.819626 4018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:07.821389 4018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:07.821636 4018 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:07.821795 4018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:14:07.824827 4018 server_base.cc:1047] running on GCE node
W20251026 15:14:07.824923 4026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:07.824810 4024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:07.824810 4023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:07.825331 4018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:07.825567 4018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:07.826790 4018 hybrid_clock.cc:648] HybridClock initialized: now 1761491647826774 us; error 33 us; skew 500 ppm
Oct 26 15:14:07 dist-test-slave-8pgz krb5kdc[4002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:07.829634 4018 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251026 15:14:07.830295 4018 webserver.cc:492] Webserver started at http://127.0.0.1:36919/ using document root <none> and password file <none>
I20251026 15:14:07.830538 4018 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:07.830585 4018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:07.830709 4018 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:07.831902 4018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "375ad75dd64641c38b92c6723a727bb8"
format_stamp: "Formatted at 2025-10-26 15:14:07 on dist-test-slave-8pgz"
server_key: "d712e2b5bdf7cd10d4a5541a66542eec"
server_key_iv: "7a95ee1b432cdc166d6c6c8e6dff2ffb"
server_key_version: "encryptionkey@0"
I20251026 15:14:07.832252 4018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "375ad75dd64641c38b92c6723a727bb8"
format_stamp: "Formatted at 2025-10-26 15:14:07 on dist-test-slave-8pgz"
server_key: "d712e2b5bdf7cd10d4a5541a66542eec"
server_key_iv: "7a95ee1b432cdc166d6c6c8e6dff2ffb"
server_key_version: "encryptionkey@0"
I20251026 15:14:07.833614 4018 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:14:07.834494 4033 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:07.834707 4018 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:14:07.834807 4018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "375ad75dd64641c38b92c6723a727bb8"
format_stamp: "Formatted at 2025-10-26 15:14:07 on dist-test-slave-8pgz"
server_key: "d712e2b5bdf7cd10d4a5541a66542eec"
server_key_iv: "7a95ee1b432cdc166d6c6c8e6dff2ffb"
server_key_version: "encryptionkey@0"
I20251026 15:14:07.834877 4018 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:07.848157 4018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:07.850759 4018 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:07.850924 4018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:07.855089 4018 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34625
I20251026 15:14:07.855193 4085 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:34625 every 8 connection(s)
I20251026 15:14:07.855459 4018 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:07.856144 4086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:07.858752 4086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8: Bootstrap starting.
I20251026 15:14:07.859302 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 4018
I20251026 15:14:07.859426 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:07.859465 4086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:07.859622 15185 external_mini_cluster.cc:1468] Setting key fd38c89f97dde73afe8f7e304c7e04c6
I20251026 15:14:07.859761 4086 log.cc:826] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:07.860605 4086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8: No bootstrap required, opened a new log
I20251026 15:14:07.861914 4086 raft_consensus.cc:359] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375ad75dd64641c38b92c6723a727bb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34625 } }
I20251026 15:14:07.862044 4086 raft_consensus.cc:385] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:07.862080 4086 raft_consensus.cc:740] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375ad75dd64641c38b92c6723a727bb8, State: Initialized, Role: FOLLOWER
I20251026 15:14:07.862185 4086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375ad75dd64641c38b92c6723a727bb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34625 } }
I20251026 15:14:07.862254 4086 raft_consensus.cc:399] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:07.862296 4086 raft_consensus.cc:493] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:07.862349 4086 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:07.862963 4086 raft_consensus.cc:515] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375ad75dd64641c38b92c6723a727bb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34625 } }
I20251026 15:14:07.863068 4086 leader_election.cc:304] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 375ad75dd64641c38b92c6723a727bb8; no voters:
I20251026 15:14:07.863377 4086 leader_election.cc:290] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:07.863559 4092 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:07.863636 4086 sys_catalog.cc:565] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:07.863785 4092 raft_consensus.cc:697] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [term 1 LEADER]: Becoming Leader. State: Replica: 375ad75dd64641c38b92c6723a727bb8, State: Running, Role: LEADER
I20251026 15:14:07.863934 4092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375ad75dd64641c38b92c6723a727bb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34625 } }
I20251026 15:14:07.864326 4091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "375ad75dd64641c38b92c6723a727bb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375ad75dd64641c38b92c6723a727bb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34625 } } }
I20251026 15:14:07.864426 4091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:07.864620 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:07.860177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:59687 (local address 127.0.0.1:34625)
1026 15:14:07.860333 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:07.860348 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:07.860376 (+ 28us) server_negotiation.cc:408] Connection header received
1026 15:14:07.860776 (+ 400us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:07.860783 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:07.860831 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:07.860956 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:07.861411 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:07.862332 (+ 921us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:07.862975 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:07.863244 (+ 269us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:07.863865 (+ 621us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:07.863874 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:07.863875 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:07.864091 (+ 216us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:07.864111 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:07.864118 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:07.864211 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:07.864411 (+ 200us) server_negotiation.cc:300] Negotiation successful
1026 15:14:07.864440 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
I20251026 15:14:07.864681 4093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 375ad75dd64641c38b92c6723a727bb8. Latest consensus state: current_term: 1 leader_uuid: "375ad75dd64641c38b92c6723a727bb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375ad75dd64641c38b92c6723a727bb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34625 } } }
I20251026 15:14:07.864732 4093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:07.864833 4100 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:07.865540 4100 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:07.865813 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:07.867520 4100 catalog_manager.cc:1357] Generated new cluster ID: 86a901cb0ddf4e63b9dd5d6f1ba09410
I20251026 15:14:07.867589 4100 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:07.870616 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:07.867395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:47003 (local address 127.0.0.1:34625)
1026 15:14:07.867434 (+ 39us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:07.867436 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:07.867694 (+ 258us) server_negotiation.cc:408] Connection header received
1026 15:14:07.867787 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:07.867788 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:07.867799 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:07.867849 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:07.868177 (+ 328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:07.868748 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:07.869448 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:07.869649 (+ 201us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:07.870256 (+ 607us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:07.870261 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:07.870262 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:07.870268 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:07.870287 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:07.870291 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:07.870360 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:07.870518 (+ 158us) server_negotiation.cc:300] Negotiation successful
1026 15:14:07.870531 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20251026 15:14:07.883426 4100 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:07.884027 4100 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:07.890278 4100 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 375ad75dd64641c38b92c6723a727bb8: Generated new TSK 0
I20251026 15:14:07.890528 4100 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:07.893676 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 4018
W20251026 15:14:07.898587 4000 connection.cc:537] client connection to 127.0.0.1:34625 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34625): Connection reset by peer (error 104)
2025-10-26T15:14:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:07 dist-test-slave-8pgz krb5kdc[4117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:07 dist-test-slave-8pgz krb5kdc[4117](info): set up 2 sockets
Oct 26 15:14:07 dist-test-slave-8pgz krb5kdc[4117](info): commencing operation
krb5kdc: starting...
W20251026 15:14:09.884006 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.971s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:09 dist-test-slave-8pgz krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:10.026100 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:33291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:10.106647 4133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:10.106859 4133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:10.106899 4133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:10.108420 4133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:10.108477 4133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:10.108501 4133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:10.108525 4133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:10.108543 4133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:10.110261 4133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:10.110555 4133 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:10.110749 4133 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:10.113447 4138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:10.113447 4141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:10.113507 4133 server_base.cc:1047] running on GCE node
W20251026 15:14:10.113442 4139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:10.113888 4133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:10.114148 4133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:10.115299 4133 hybrid_clock.cc:648] HybridClock initialized: now 1761491650115278 us; error 43 us; skew 500 ppm
Oct 26 15:14:10 dist-test-slave-8pgz krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:10.117936 4133 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251026 15:14:10.118566 4133 webserver.cc:492] Webserver started at http://127.0.0.1:40765/ using document root <none> and password file <none>
I20251026 15:14:10.118757 4133 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:10.118798 4133 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:10.118917 4133 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:10.119894 4133 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "399da67de658476d9ecc907a82214594"
format_stamp: "Formatted at 2025-10-26 15:14:10 on dist-test-slave-8pgz"
server_key: "1ad777bb81d81e2c09754a4e9adb4548"
server_key_iv: "b52819e261e1844ded892c6cb52abdf8"
server_key_version: "encryptionkey@0"
I20251026 15:14:10.120214 4133 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "399da67de658476d9ecc907a82214594"
format_stamp: "Formatted at 2025-10-26 15:14:10 on dist-test-slave-8pgz"
server_key: "1ad777bb81d81e2c09754a4e9adb4548"
server_key_iv: "b52819e261e1844ded892c6cb52abdf8"
server_key_version: "encryptionkey@0"
I20251026 15:14:10.121454 4133 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:14:10.122184 4148 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:10.122385 4133 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:14:10.122495 4133 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "399da67de658476d9ecc907a82214594"
format_stamp: "Formatted at 2025-10-26 15:14:10 on dist-test-slave-8pgz"
server_key: "1ad777bb81d81e2c09754a4e9adb4548"
server_key_iv: "b52819e261e1844ded892c6cb52abdf8"
server_key_version: "encryptionkey@0"
I20251026 15:14:10.122570 4133 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:10.125880 4133 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:10.131330 4133 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:10.131448 4133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:10.135340 4133 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45533
I20251026 15:14:10.135375 4200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:45533 every 8 connection(s)
I20251026 15:14:10.135704 4133 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:10.136343 4201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:10.138896 4201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594: Bootstrap starting.
I20251026 15:14:10.139545 4201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:10.139786 4201 log.cc:826] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:10.140587 4201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594: No bootstrap required, opened a new log
I20251026 15:14:10.140652 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 4133
I20251026 15:14:10.140759 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:10.141008 15185 external_mini_cluster.cc:1468] Setting key 30fd5d91abf23406235f6064b0f16f62
I20251026 15:14:10.142289 4201 raft_consensus.cc:359] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399da67de658476d9ecc907a82214594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45533 } }
I20251026 15:14:10.142498 4201 raft_consensus.cc:385] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:10.142573 4201 raft_consensus.cc:740] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 399da67de658476d9ecc907a82214594, State: Initialized, Role: FOLLOWER
I20251026 15:14:10.142715 4201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399da67de658476d9ecc907a82214594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45533 } }
I20251026 15:14:10.142808 4201 raft_consensus.cc:399] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:10.142867 4201 raft_consensus.cc:493] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:10.142925 4201 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:10.143702 4201 raft_consensus.cc:515] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399da67de658476d9ecc907a82214594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45533 } }
I20251026 15:14:10.143761 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:10.141599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:52961 (local address 127.0.0.1:45533)
1026 15:14:10.141875 (+ 276us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:10.141885 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:10.141909 (+ 24us) server_negotiation.cc:408] Connection header received
1026 15:14:10.142271 (+ 362us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:10.142280 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:10.142313 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:10.142563 (+ 250us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:10.143308 (+ 745us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:10.143312 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:10.143312 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:10.143317 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:10.143341 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:10.143349 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:10.143444 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:10.143542 (+ 98us) server_negotiation.cc:300] Negotiation successful
1026 15:14:10.143581 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20251026 15:14:10.143832 4201 leader_election.cc:304] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 399da67de658476d9ecc907a82214594; no voters:
I20251026 15:14:10.143970 4201 leader_election.cc:290] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:10.144029 4206 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:10.144153 4206 raft_consensus.cc:697] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [term 1 LEADER]: Becoming Leader. State: Replica: 399da67de658476d9ecc907a82214594, State: Running, Role: LEADER
I20251026 15:14:10.144220 4201 sys_catalog.cc:565] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:10.144259 4206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399da67de658476d9ecc907a82214594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45533 } }
I20251026 15:14:10.144562 4206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "399da67de658476d9ecc907a82214594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399da67de658476d9ecc907a82214594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45533 } } }
I20251026 15:14:10.144639 4206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:10.144760 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:10.144744 4206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 399da67de658476d9ecc907a82214594. Latest consensus state: current_term: 1 leader_uuid: "399da67de658476d9ecc907a82214594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399da67de658476d9ecc907a82214594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45533 } } }
I20251026 15:14:10.144815 4206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:10.144984 4210 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:10.145493 4210 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:10.148046 4210 catalog_manager.cc:1357] Generated new cluster ID: cd6aa89647fc42eb89ecde53b1046c06
I20251026 15:14:10.148123 4210 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:10.156435 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:10.146764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:42737 (local address 127.0.0.1:45533)
1026 15:14:10.146851 (+ 87us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:10.146852 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:10.147096 (+ 244us) server_negotiation.cc:408] Connection header received
1026 15:14:10.147193 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:10.147194 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:10.147199 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:10.147257 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:10.147972 (+ 715us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:10.147974 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:10.147974 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:10.147980 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:10.148000 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:10.148005 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:10.148069 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:10.156312 (+ 8243us) server_negotiation.cc:300] Negotiation successful
1026 15:14:10.156334 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20251026 15:14:10.156809 4210 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:10.156987 4210 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:10.167490 4210 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 399da67de658476d9ecc907a82214594: Generated new TSK 0
I20251026 15:14:10.167735 4210 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:10.173188 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 4133
2025-10-26T15:14:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:10 dist-test-slave-8pgz krb5kdc[4232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:10 dist-test-slave-8pgz krb5kdc[4232](info): set up 2 sockets
Oct 26 15:14:10 dist-test-slave-8pgz krb5kdc[4232](info): commencing operation
krb5kdc: starting...
W20251026 15:14:12.171958 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.980s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:12 dist-test-slave-8pgz krb5kdc[4232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:12.312604 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:12.393713 4248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:12.393939 4248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:12.393965 4248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:12.395577 4248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:12.395639 4248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:12.395654 4248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:12.395668 4248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:12.395694 4248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:12.397514 4248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:12.397796 4248 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:12.398000 4248 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:12.400717 4254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:12.400736 4253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:12.400717 4256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:12.401017 4248 server_base.cc:1047] running on GCE node
I20251026 15:14:12.401261 4248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:12.401512 4248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:12.402701 4248 hybrid_clock.cc:648] HybridClock initialized: now 1761491652402685 us; error 30 us; skew 500 ppm
Oct 26 15:14:12 dist-test-slave-8pgz krb5kdc[4232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:12.405421 4248 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251026 15:14:12.405970 4248 webserver.cc:492] Webserver started at http://127.0.0.1:35593/ using document root <none> and password file <none>
I20251026 15:14:12.406162 4248 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:12.406200 4248 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:12.406323 4248 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:12.407342 4248 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "c31acffdf80b48b7abef965326a19960"
format_stamp: "Formatted at 2025-10-26 15:14:12 on dist-test-slave-8pgz"
server_key: "a7a24a4bdec5870cb463aa4231cacdf8"
server_key_iv: "04e79fd392934ece4c579f32be50ee76"
server_key_version: "encryptionkey@0"
I20251026 15:14:12.407663 4248 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "c31acffdf80b48b7abef965326a19960"
format_stamp: "Formatted at 2025-10-26 15:14:12 on dist-test-slave-8pgz"
server_key: "a7a24a4bdec5870cb463aa4231cacdf8"
server_key_iv: "04e79fd392934ece4c579f32be50ee76"
server_key_version: "encryptionkey@0"
I20251026 15:14:12.408969 4248 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:14:12.409883 4263 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:12.410218 4248 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:14:12.410310 4248 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "c31acffdf80b48b7abef965326a19960"
format_stamp: "Formatted at 2025-10-26 15:14:12 on dist-test-slave-8pgz"
server_key: "a7a24a4bdec5870cb463aa4231cacdf8"
server_key_iv: "04e79fd392934ece4c579f32be50ee76"
server_key_version: "encryptionkey@0"
I20251026 15:14:12.410387 4248 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:12.422048 4248 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:12.424090 4248 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:12.424243 4248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:12.428181 4248 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34323
I20251026 15:14:12.428216 4315 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:34323 every 8 connection(s)
I20251026 15:14:12.428552 4248 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:12.429143 4316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:12.431777 4316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960: Bootstrap starting.
I20251026 15:14:12.432412 4316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:12.432658 4316 log.cc:826] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:12.433399 4316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960: No bootstrap required, opened a new log
I20251026 15:14:12.434674 4316 raft_consensus.cc:359] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31acffdf80b48b7abef965326a19960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20251026 15:14:12.434808 4316 raft_consensus.cc:385] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:12.434881 4316 raft_consensus.cc:740] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31acffdf80b48b7abef965326a19960, State: Initialized, Role: FOLLOWER
I20251026 15:14:12.435019 4316 consensus_queue.cc:260] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31acffdf80b48b7abef965326a19960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20251026 15:14:12.435101 4316 raft_consensus.cc:399] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:12.435142 4316 raft_consensus.cc:493] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:12.435194 4316 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:12.435736 4316 raft_consensus.cc:515] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31acffdf80b48b7abef965326a19960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20251026 15:14:12.435846 4316 leader_election.cc:304] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c31acffdf80b48b7abef965326a19960; no voters:
I20251026 15:14:12.435997 4316 leader_election.cc:290] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:12.436046 4319 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:12.436197 4319 raft_consensus.cc:697] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [term 1 LEADER]: Becoming Leader. State: Replica: c31acffdf80b48b7abef965326a19960, State: Running, Role: LEADER
I20251026 15:14:12.436234 4316 sys_catalog.cc:565] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:12.436343 4319 consensus_queue.cc:237] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31acffdf80b48b7abef965326a19960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20251026 15:14:12.436671 4320 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c31acffdf80b48b7abef965326a19960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31acffdf80b48b7abef965326a19960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } } }
I20251026 15:14:12.436707 4321 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31acffdf80b48b7abef965326a19960. Latest consensus state: current_term: 1 leader_uuid: "c31acffdf80b48b7abef965326a19960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31acffdf80b48b7abef965326a19960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } } }
I20251026 15:14:12.436786 4320 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:12.436805 4321 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:12.436998 4325 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:12.437367 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 4248
I20251026 15:14:12.437472 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:12.437675 4325 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:12.437829 15185 external_mini_cluster.cc:1468] Setting key 8d886061f4efad269e4980681be0e7d2
I20251026 15:14:12.439559 4325 catalog_manager.cc:1357] Generated new cluster ID: 1e81ca0f0daa4b02bf7f89b7a0691bb9
I20251026 15:14:12.439616 4325 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 26 15:14:12 dist-test-slave-8pgz krb5kdc[4232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20251026 15:14:12.447290 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:12.438536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:52547 (local address 127.0.0.1:34323)
1026 15:14:12.438949 (+ 413us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:12.438961 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:12.438985 (+ 24us) server_negotiation.cc:408] Connection header received
1026 15:14:12.439092 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:12.439097 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:12.439150 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:12.439280 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:12.440146 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:12.440926 (+ 780us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:12.441486 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:12.442143 (+ 657us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:12.443955 (+ 1812us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:12.443963 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:12.443963 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:12.443991 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:12.445716 (+ 1725us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:12.446130 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:12.446132 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:12.446132 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:12.446191 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:12.446381 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:12.446382 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:12.446382 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:12.446667 (+ 285us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:12.446833 (+ 166us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:12.446990 (+ 157us) server_negotiation.cc:300] Negotiation successful
1026 15:14:12.447119 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":64,"threads_started":1}
I20251026 15:14:12.448323 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:12.448617 4325 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:12.449333 4325 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:12.454982 4325 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c31acffdf80b48b7abef965326a19960: Generated new TSK 0
I20251026 15:14:12.455205 4325 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:12.460611 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:12.450050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:55151 (local address 127.0.0.1:34323)
1026 15:14:12.450085 (+ 35us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:12.450087 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:12.454630 (+ 4543us) server_negotiation.cc:408] Connection header received
1026 15:14:12.454700 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:12.454701 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:12.454715 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:12.454765 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:12.455616 (+ 851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:12.456117 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:12.456676 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:12.456881 (+ 205us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:12.458129 (+ 1248us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:12.458134 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:12.458134 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:12.458142 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:12.459584 (+ 1442us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:12.459935 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:12.459936 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:12.459937 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:12.459957 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:12.460121 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:12.460122 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:12.460123 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:12.460212 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:12.460311 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:12.460439 (+ 128us) server_negotiation.cc:300] Negotiation successful
1026 15:14:12.460507 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20251026 15:14:12.462854 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 4248
W20251026 15:14:12.467396 4230 connection.cc:537] client connection to 127.0.0.1:34323 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34323): Connection reset by peer (error 104)
2025-10-26T15:14:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:12 dist-test-slave-8pgz krb5kdc[4347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:12 dist-test-slave-8pgz krb5kdc[4347](info): set up 2 sockets
Oct 26 15:14:12 dist-test-slave-8pgz krb5kdc[4347](info): commencing operation
krb5kdc: starting...
W20251026 15:14:14.459997 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.979s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:14 dist-test-slave-8pgz krb5kdc[4347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:14.599370 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:42177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:14.683849 4363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:14.684080 4363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:14.684124 4363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:14.685679 4363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:14.685753 4363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:14.685792 4363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:14.685814 4363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:14.685838 4363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:14.687680 4363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41655
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:14.687990 4363 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:14.688196 4363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:14.690971 4369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:14.690969 4371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:14.691107 4363 server_base.cc:1047] running on GCE node
W20251026 15:14:14.691221 4368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:14.691419 4363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:14.691659 4363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:14.692790 4363 hybrid_clock.cc:648] HybridClock initialized: now 1761491654692778 us; error 31 us; skew 500 ppm
Oct 26 15:14:14 dist-test-slave-8pgz krb5kdc[4347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:14.695484 4363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251026 15:14:14.696061 4363 webserver.cc:492] Webserver started at http://127.0.0.1:45809/ using document root <none> and password file <none>
I20251026 15:14:14.696262 4363 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:14.696301 4363 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:14.696398 4363 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:14.697431 4363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "7c922b3d5bf3407c9e489b1f168ee08a"
format_stamp: "Formatted at 2025-10-26 15:14:14 on dist-test-slave-8pgz"
server_key: "dddebc6d534848a10c122124fb05328a"
server_key_iv: "f5458afcaba756f86e6511dfd311802a"
server_key_version: "encryptionkey@0"
I20251026 15:14:14.697779 4363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "7c922b3d5bf3407c9e489b1f168ee08a"
format_stamp: "Formatted at 2025-10-26 15:14:14 on dist-test-slave-8pgz"
server_key: "dddebc6d534848a10c122124fb05328a"
server_key_iv: "f5458afcaba756f86e6511dfd311802a"
server_key_version: "encryptionkey@0"
I20251026 15:14:14.699121 4363 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:14:14.699843 4378 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:14.700017 4363 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:14.700089 4363 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "7c922b3d5bf3407c9e489b1f168ee08a"
format_stamp: "Formatted at 2025-10-26 15:14:14 on dist-test-slave-8pgz"
server_key: "dddebc6d534848a10c122124fb05328a"
server_key_iv: "f5458afcaba756f86e6511dfd311802a"
server_key_version: "encryptionkey@0"
I20251026 15:14:14.700161 4363 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:14.711424 4363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:14.713485 4363 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:14.713624 4363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:14.717700 4363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41655
I20251026 15:14:14.717757 4430 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:41655 every 8 connection(s)
I20251026 15:14:14.718103 4363 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:14.718804 4431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:14.721552 4431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a: Bootstrap starting.
I20251026 15:14:14.722198 4431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:14.722478 4431 log.cc:826] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:14.723223 4431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a: No bootstrap required, opened a new log
I20251026 15:14:14.724182 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 4363
I20251026 15:14:14.724284 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:14.724444 15185 external_mini_cluster.cc:1468] Setting key f7f496477962628b26380b0ed12f18a0
I20251026 15:14:14.724617 4431 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c922b3d5bf3407c9e489b1f168ee08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41655 } }
I20251026 15:14:14.724800 4431 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:14.724877 4431 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c922b3d5bf3407c9e489b1f168ee08a, State: Initialized, Role: FOLLOWER
I20251026 15:14:14.725050 4431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c922b3d5bf3407c9e489b1f168ee08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41655 } }
I20251026 15:14:14.725168 4431 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:14.725253 4431 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:14.725307 4431 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:14.726082 4431 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c922b3d5bf3407c9e489b1f168ee08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41655 } }
I20251026 15:14:14.726207 4431 leader_election.cc:304] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c922b3d5bf3407c9e489b1f168ee08a; no voters:
I20251026 15:14:14.726368 4431 leader_election.cc:290] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:14.726459 4436 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:14.726637 4431 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:14.726637 4436 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [term 1 LEADER]: Becoming Leader. State: Replica: 7c922b3d5bf3407c9e489b1f168ee08a, State: Running, Role: LEADER
I20251026 15:14:14.726879 4436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c922b3d5bf3407c9e489b1f168ee08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41655 } }
I20251026 15:14:14.727321 4437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c922b3d5bf3407c9e489b1f168ee08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c922b3d5bf3407c9e489b1f168ee08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41655 } } }
I20251026 15:14:14.727399 4437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:14.727847 4438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c922b3d5bf3407c9e489b1f168ee08a. Latest consensus state: current_term: 1 leader_uuid: "7c922b3d5bf3407c9e489b1f168ee08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c922b3d5bf3407c9e489b1f168ee08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41655 } } }
I20251026 15:14:14.728178 4438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:14.728417 4445 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:14.729038 4445 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:14.729555 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:14.725046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:60909 (local address 127.0.0.1:41655)
1026 15:14:14.725266 (+ 220us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:14.725276 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:14.725300 (+ 24us) server_negotiation.cc:408] Connection header received
1026 15:14:14.725415 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:14.725419 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:14.725462 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:14.725582 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:14.726041 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:14.727143 (+ 1102us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:14.728216 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:14.728466 (+ 250us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:14.728794 (+ 328us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:14.728801 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:14.728801 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:14.729026 (+ 225us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:14.729048 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:14.729057 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:14.729219 (+ 162us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:14.729356 (+ 137us) server_negotiation.cc:300] Negotiation successful
1026 15:14:14.729388 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
I20251026 15:14:14.730506 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:14.730680 4445 catalog_manager.cc:1357] Generated new cluster ID: d147f5d944e74165a2e22852d251b91c
I20251026 15:14:14.730728 4445 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:14.735028 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:14.731986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:33883 (local address 127.0.0.1:41655)
1026 15:14:14.732027 (+ 41us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:14.732029 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:14.732211 (+ 182us) server_negotiation.cc:408] Connection header received
1026 15:14:14.732258 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:14.732259 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:14.732271 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:14.732351 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:14.732673 (+ 322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:14.733376 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:14.734079 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:14.734266 (+ 187us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:14.734758 (+ 492us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:14.734762 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:14.734762 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:14.734767 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:14.734783 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:14.734787 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:14.734862 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:14.734930 (+ 68us) server_negotiation.cc:300] Negotiation successful
1026 15:14:14.734943 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251026 15:14:14.739652 4445 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:14.740213 4445 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:14.744163 4445 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7c922b3d5bf3407c9e489b1f168ee08a: Generated new TSK 0
I20251026 15:14:14.744450 4445 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:14.752074 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 4363
W20251026 15:14:14.756544 4345 connection.cc:537] client connection to 127.0.0.1:41655 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41655): Connection reset by peer (error 104)
2025-10-26T15:14:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:14 dist-test-slave-8pgz krb5kdc[4462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:14 dist-test-slave-8pgz krb5kdc[4462](info): set up 2 sockets
Oct 26 15:14:14 dist-test-slave-8pgz krb5kdc[4462](info): commencing operation
krb5kdc: starting...
W20251026 15:14:16.767927 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.997s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:16 dist-test-slave-8pgz krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:16.909535 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:16.989465 4478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:16.989675 4478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:16.989715 4478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:16.991212 4478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:16.991266 4478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:16.991289 4478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:16.991307 4478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:16.991328 4478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:16.992996 4478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38651
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:16.993237 4478 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:16.993422 4478 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:16.996003 4483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:16.995966 4486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:16.995966 4484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:16.996203 4478 server_base.cc:1047] running on GCE node
I20251026 15:14:16.996462 4478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:16.996681 4478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:16.997831 4478 hybrid_clock.cc:648] HybridClock initialized: now 1761491656997823 us; error 33 us; skew 500 ppm
Oct 26 15:14:16 dist-test-slave-8pgz krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:17.000340 4478 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251026 15:14:17.000908 4478 webserver.cc:492] Webserver started at http://127.0.0.1:40969/ using document root <none> and password file <none>
I20251026 15:14:17.001120 4478 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:17.001161 4478 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:17.001272 4478 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:17.002204 4478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "63011c10491c4547a57e2d4ed4b162a6"
format_stamp: "Formatted at 2025-10-26 15:14:16 on dist-test-slave-8pgz"
server_key: "3b893829955e36c9c86b30e149006b39"
server_key_iv: "dc1295833abe5d5b462c8b294d231e25"
server_key_version: "encryptionkey@0"
I20251026 15:14:17.002560 4478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "63011c10491c4547a57e2d4ed4b162a6"
format_stamp: "Formatted at 2025-10-26 15:14:16 on dist-test-slave-8pgz"
server_key: "3b893829955e36c9c86b30e149006b39"
server_key_iv: "dc1295833abe5d5b462c8b294d231e25"
server_key_version: "encryptionkey@0"
I20251026 15:14:17.003758 4478 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:14:17.004604 4493 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:17.004806 4478 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:14:17.004889 4478 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "63011c10491c4547a57e2d4ed4b162a6"
format_stamp: "Formatted at 2025-10-26 15:14:16 on dist-test-slave-8pgz"
server_key: "3b893829955e36c9c86b30e149006b39"
server_key_iv: "dc1295833abe5d5b462c8b294d231e25"
server_key_version: "encryptionkey@0"
I20251026 15:14:17.004951 4478 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:17.008381 4478 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:17.014737 4478 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:17.014904 4478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:17.019269 4478 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38651
I20251026 15:14:17.019313 4545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:38651 every 8 connection(s)
I20251026 15:14:17.019680 4478 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:17.020330 4546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:17.023003 4546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6: Bootstrap starting.
I20251026 15:14:17.023600 4546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:17.023846 4546 log.cc:826] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:17.024255 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 4478
I20251026 15:14:17.024410 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:17.024573 15185 external_mini_cluster.cc:1468] Setting key 11a31203bf741ce3e2411acb632a4113
I20251026 15:14:17.024753 4546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6: No bootstrap required, opened a new log
I20251026 15:14:17.026659 4546 raft_consensus.cc:359] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63011c10491c4547a57e2d4ed4b162a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38651 } }
I20251026 15:14:17.026830 4546 raft_consensus.cc:385] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:17.026868 4546 raft_consensus.cc:740] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63011c10491c4547a57e2d4ed4b162a6, State: Initialized, Role: FOLLOWER
I20251026 15:14:17.026968 4546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63011c10491c4547a57e2d4ed4b162a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38651 } }
I20251026 15:14:17.027038 4546 raft_consensus.cc:399] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:17.027070 4546 raft_consensus.cc:493] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:17.027149 4546 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:17.027284 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:17.025089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:49259 (local address 127.0.0.1:38651)
1026 15:14:17.025251 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:17.025268 (+ 17us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:17.025294 (+ 26us) server_negotiation.cc:408] Connection header received
1026 15:14:17.025866 (+ 572us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:17.025875 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:17.025912 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:17.026067 (+ 155us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:17.026724 (+ 657us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:17.026726 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:17.026726 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:17.026731 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:17.026755 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:17.026762 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:17.026859 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:17.027077 (+ 218us) server_negotiation.cc:300] Negotiation successful
1026 15:14:17.027119 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20251026 15:14:17.027968 4546 raft_consensus.cc:515] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63011c10491c4547a57e2d4ed4b162a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38651 } }
I20251026 15:14:17.028110 4546 leader_election.cc:304] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63011c10491c4547a57e2d4ed4b162a6; no voters:
I20251026 15:14:17.028273 4546 leader_election.cc:290] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:17.028278 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:17.028302 4551 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:17.028409 4551 raft_consensus.cc:697] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [term 1 LEADER]: Becoming Leader. State: Replica: 63011c10491c4547a57e2d4ed4b162a6, State: Running, Role: LEADER
I20251026 15:14:17.028529 4546 sys_catalog.cc:565] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:17.028556 4551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63011c10491c4547a57e2d4ed4b162a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38651 } }
I20251026 15:14:17.028942 4553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63011c10491c4547a57e2d4ed4b162a6. Latest consensus state: current_term: 1 leader_uuid: "63011c10491c4547a57e2d4ed4b162a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63011c10491c4547a57e2d4ed4b162a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38651 } } }
I20251026 15:14:17.029011 4553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:17.029223 4552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63011c10491c4547a57e2d4ed4b162a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63011c10491c4547a57e2d4ed4b162a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38651 } } }
I20251026 15:14:17.029874 4552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6 [sys.catalog]: This master's current role is: LEADER
W20251026 15:14:17.030625 4571 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:14:17.030731 4571 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:14:17.030800 4555 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:17.031038 4555 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:17.031998 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:17.030569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:35269 (local address 127.0.0.1:38651)
1026 15:14:17.030613 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:17.030615 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:17.030813 (+ 198us) server_negotiation.cc:408] Connection header received
1026 15:14:17.030925 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:17.030927 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:17.030932 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:17.030962 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:17.031775 (+ 813us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:17.031776 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:17.031776 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:17.031780 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:17.031796 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:17.031800 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:17.031866 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:17.031917 (+ 51us) server_negotiation.cc:300] Negotiation successful
1026 15:14:17.031933 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251026 15:14:17.032649 4555 catalog_manager.cc:1357] Generated new cluster ID: 2008e01aa69c4501b5fc92639750b496
I20251026 15:14:17.032699 4555 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:17.037179 4555 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:17.037338 4555 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:17.045428 4555 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 63011c10491c4547a57e2d4ed4b162a6: Generated new TSK 0
I20251026 15:14:17.045639 4555 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:17.052130 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 4478
2025-10-26T15:14:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:17 dist-test-slave-8pgz krb5kdc[4577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:17 dist-test-slave-8pgz krb5kdc[4577](info): set up 2 sockets
Oct 26 15:14:17 dist-test-slave-8pgz krb5kdc[4577](info): commencing operation
krb5kdc: starting...
W20251026 15:14:19.040206 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.968s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:19 dist-test-slave-8pgz krb5kdc[4577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:19.182159 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:39203
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:39203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:19.266777 4593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:19.267019 4593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:19.267047 4593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:19.268692 4593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:19.268777 4593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:19.268795 4593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:19.268810 4593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:19.268826 4593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:19.270736 4593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:39203
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:39203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:19.271016 4593 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:19.271234 4593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:19.274278 4601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:19.274312 4598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:19.274302 4599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:19.274502 4593 server_base.cc:1047] running on GCE node
I20251026 15:14:19.274732 4593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:19.274989 4593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:19.276180 4593 hybrid_clock.cc:648] HybridClock initialized: now 1761491659276157 us; error 32 us; skew 500 ppm
Oct 26 15:14:19 dist-test-slave-8pgz krb5kdc[4577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491659, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:19.279174 4593 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:14:19.279736 4593 webserver.cc:492] Webserver started at http://127.14.212.126:39645/ using document root <none> and password file <none>
I20251026 15:14:19.279942 4593 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:19.279981 4593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:19.280102 4593 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:19.281105 4593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "3c8f42deae534b95ae1049c640ff1f0a"
format_stamp: "Formatted at 2025-10-26 15:14:19 on dist-test-slave-8pgz"
server_key: "5e828b3dc71ef4f61d30023205c7b8e1"
server_key_iv: "8b0311a396d60ee2ef3401724721c114"
server_key_version: "encryptionkey@0"
I20251026 15:14:19.281458 4593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "3c8f42deae534b95ae1049c640ff1f0a"
format_stamp: "Formatted at 2025-10-26 15:14:19 on dist-test-slave-8pgz"
server_key: "5e828b3dc71ef4f61d30023205c7b8e1"
server_key_iv: "8b0311a396d60ee2ef3401724721c114"
server_key_version: "encryptionkey@0"
I20251026 15:14:19.282846 4593 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:14:19.283670 4608 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:19.283871 4593 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:14:19.283953 4593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "3c8f42deae534b95ae1049c640ff1f0a"
format_stamp: "Formatted at 2025-10-26 15:14:19 on dist-test-slave-8pgz"
server_key: "5e828b3dc71ef4f61d30023205c7b8e1"
server_key_iv: "8b0311a396d60ee2ef3401724721c114"
server_key_version: "encryptionkey@0"
I20251026 15:14:19.284030 4593 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:19.307150 4593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:19.309393 4593 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:19.309540 4593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:19.313586 4593 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:39203
I20251026 15:14:19.313620 4660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:39203 every 8 connection(s)
I20251026 15:14:19.313969 4593 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:19.314646 4661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:19.317255 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 4593
I20251026 15:14:19.317286 4661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a: Bootstrap starting.
I20251026 15:14:19.317390 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:19.317567 15185 external_mini_cluster.cc:1468] Setting key 74a8a117ed34dedc371a28182fed92cb
I20251026 15:14:19.318051 4661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:19.318463 4661 log.cc:826] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:19.319216 4661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a: No bootstrap required, opened a new log
I20251026 15:14:19.320580 4661 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f42deae534b95ae1049c640ff1f0a" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39203 } }
I20251026 15:14:19.320710 4661 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:19.320731 4661 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8f42deae534b95ae1049c640ff1f0a, State: Initialized, Role: FOLLOWER
I20251026 15:14:19.320858 4661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f42deae534b95ae1049c640ff1f0a" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39203 } }
I20251026 15:14:19.320927 4661 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:19.320958 4661 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:19.321034 4661 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:19.321868 4661 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f42deae534b95ae1049c640ff1f0a" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39203 } }
I20251026 15:14:19.322002 4661 leader_election.cc:304] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c8f42deae534b95ae1049c640ff1f0a; no voters:
I20251026 15:14:19.322185 4661 leader_election.cc:290] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:19.322256 4666 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:19.322466 4666 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [term 1 LEADER]: Becoming Leader. State: Replica: 3c8f42deae534b95ae1049c640ff1f0a, State: Running, Role: LEADER
I20251026 15:14:19.322472 4661 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:19.322644 4666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f42deae534b95ae1049c640ff1f0a" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39203 } }
I20251026 15:14:19.323412 4668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c8f42deae534b95ae1049c640ff1f0a. Latest consensus state: current_term: 1 leader_uuid: "3c8f42deae534b95ae1049c640ff1f0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f42deae534b95ae1049c640ff1f0a" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39203 } } }
I20251026 15:14:19.323534 4668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:19.323756 4676 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
Oct 26 15:14:19 dist-test-slave-8pgz krb5kdc[4577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:14:19.324359 4676 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:19.324518 4667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c8f42deae534b95ae1049c640ff1f0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f42deae534b95ae1049c640ff1f0a" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39203 } } }
I20251026 15:14:19.324596 4667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:19.326385 4676 catalog_manager.cc:1357] Generated new cluster ID: dbaaeb344d474a22bb9ebce05a578fb6
I20251026 15:14:19.326467 4676 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:19.329340 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:19.318236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33707 (local address 127.14.212.126:39203)
1026 15:14:19.318448 (+ 212us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:19.318460 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:19.318482 (+ 22us) server_negotiation.cc:408] Connection header received
1026 15:14:19.318856 (+ 374us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:19.318863 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:19.318910 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:19.319044 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:19.320051 (+ 1007us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:19.320873 (+ 822us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:19.321674 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:19.321906 (+ 232us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:19.324786 (+ 2880us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:19.324796 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:19.324796 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:19.324822 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:19.327308 (+ 2486us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:19.327823 (+ 515us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:19.327826 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:19.327827 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:19.327896 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:19.328138 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:19.328140 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:19.328140 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:19.328397 (+ 257us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:19.328546 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:19.329021 (+ 475us) server_negotiation.cc:300] Negotiation successful
1026 15:14:19.329149 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20251026 15:14:19.336236 4676 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:19.336429 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:19.336855 4676 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:19.345512 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:19.338115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33609 (local address 127.14.212.126:39203)
1026 15:14:19.338156 (+ 41us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:19.338158 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:19.338261 (+ 103us) server_negotiation.cc:408] Connection header received
1026 15:14:19.338354 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:19.338355 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:19.338365 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:19.338446 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:19.339350 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:19.340121 (+ 771us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:19.341057 (+ 936us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:19.341240 (+ 183us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:19.342552 (+ 1312us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:19.342557 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:19.342557 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:19.342565 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:19.344338 (+ 1773us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:19.344776 (+ 438us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:19.344777 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:19.344778 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:19.344799 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:19.344992 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:19.344993 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:19.344993 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:19.345103 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:19.345202 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:19.345343 (+ 141us) server_negotiation.cc:300] Negotiation successful
1026 15:14:19.345408 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251026 15:14:19.346899 4676 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3c8f42deae534b95ae1049c640ff1f0a: Generated new TSK 0
I20251026 15:14:19.347082 4676 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:19.349074 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 4593
W20251026 15:14:19.353197 4575 connection.cc:537] client connection to 127.14.212.126:39203 recv error: Network error: failed to read from TLS socket (remote: 127.14.212.126:39203): Connection reset by peer (error 104)
2025-10-26T15:14:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:19 dist-test-slave-8pgz krb5kdc[4692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:19 dist-test-slave-8pgz krb5kdc[4692](info): set up 2 sockets
Oct 26 15:14:19 dist-test-slave-8pgz krb5kdc[4692](info): commencing operation
krb5kdc: starting...
W20251026 15:14:21.344175 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.977s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:21 dist-test-slave-8pgz krb5kdc[4692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:21.485787 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:39031
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:39031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:21.568676 4708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:21.568892 4708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:21.568917 4708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:21.570441 4708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:21.570487 4708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:21.570501 4708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:21.570514 4708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:21.570526 4708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:21.572218 4708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:39031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:39031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:21.572438 4708 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:21.572605 4708 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:14:21.575314 4708 server_base.cc:1047] running on GCE node
W20251026 15:14:21.575270 4714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:21.575285 4716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:21.575282 4713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:21.575671 4708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:21.575906 4708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:21.577100 4708 hybrid_clock.cc:648] HybridClock initialized: now 1761491661577084 us; error 28 us; skew 500 ppm
Oct 26 15:14:21 dist-test-slave-8pgz krb5kdc[4692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491661, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:21.579958 4708 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:14:21.580510 4708 webserver.cc:492] Webserver started at http://127.14.212.126:34977/ using document root <none> and password file <none>
I20251026 15:14:21.580708 4708 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:21.580750 4708 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:21.580885 4708 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:21.581864 4708 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "30d46813f6554104a3bf21acf600b29d"
format_stamp: "Formatted at 2025-10-26 15:14:21 on dist-test-slave-8pgz"
server_key: "464e77c9a07083fd97c042f110093202"
server_key_iv: "a9ffd42689a3f5301b80b5abdef4796e"
server_key_version: "encryptionkey@0"
I20251026 15:14:21.582172 4708 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "30d46813f6554104a3bf21acf600b29d"
format_stamp: "Formatted at 2025-10-26 15:14:21 on dist-test-slave-8pgz"
server_key: "464e77c9a07083fd97c042f110093202"
server_key_iv: "a9ffd42689a3f5301b80b5abdef4796e"
server_key_version: "encryptionkey@0"
I20251026 15:14:21.583491 4708 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:14:21.584211 4723 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:21.584385 4708 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:14:21.584463 4708 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "30d46813f6554104a3bf21acf600b29d"
format_stamp: "Formatted at 2025-10-26 15:14:21 on dist-test-slave-8pgz"
server_key: "464e77c9a07083fd97c042f110093202"
server_key_iv: "a9ffd42689a3f5301b80b5abdef4796e"
server_key_version: "encryptionkey@0"
I20251026 15:14:21.584525 4708 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:21.620167 4708 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:21.622459 4708 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:21.622622 4708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:21.626904 4708 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:39031
I20251026 15:14:21.626937 4775 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:39031 every 8 connection(s)
I20251026 15:14:21.627296 4708 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:21.627918 4776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:21.630474 4776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d: Bootstrap starting.
I20251026 15:14:21.630777 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 4708
I20251026 15:14:21.630913 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:21.631096 15185 external_mini_cluster.cc:1468] Setting key 6c645de38a5aa9d7bdea68db3a231828
I20251026 15:14:21.631294 4776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:21.631556 4776 log.cc:826] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:21.632337 4776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d: No bootstrap required, opened a new log
I20251026 15:14:21.633814 4776 raft_consensus.cc:359] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d46813f6554104a3bf21acf600b29d" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39031 } }
I20251026 15:14:21.633970 4776 raft_consensus.cc:385] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:21.634001 4776 raft_consensus.cc:740] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d46813f6554104a3bf21acf600b29d, State: Initialized, Role: FOLLOWER
I20251026 15:14:21.634105 4776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d46813f6554104a3bf21acf600b29d" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39031 } }
I20251026 15:14:21.634184 4776 raft_consensus.cc:399] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:21.634222 4776 raft_consensus.cc:493] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:21.634268 4776 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:21.634912 4776 raft_consensus.cc:515] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d46813f6554104a3bf21acf600b29d" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39031 } }
I20251026 15:14:21.635030 4776 leader_election.cc:304] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d46813f6554104a3bf21acf600b29d; no voters:
I20251026 15:14:21.635223 4776 leader_election.cc:290] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:21.635279 4781 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:21.635440 4781 raft_consensus.cc:697] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [term 1 LEADER]: Becoming Leader. State: Replica: 30d46813f6554104a3bf21acf600b29d, State: Running, Role: LEADER
I20251026 15:14:21.635488 4776 sys_catalog.cc:565] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:21.635540 4781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d46813f6554104a3bf21acf600b29d" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39031 } }
I20251026 15:14:21.635836 4781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d46813f6554104a3bf21acf600b29d. Latest consensus state: current_term: 1 leader_uuid: "30d46813f6554104a3bf21acf600b29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d46813f6554104a3bf21acf600b29d" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39031 } } }
I20251026 15:14:21.635849 4782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30d46813f6554104a3bf21acf600b29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d46813f6554104a3bf21acf600b29d" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 39031 } } }
I20251026 15:14:21.636030 4782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:21.636018 4781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:21.636303 4788 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:21.636756 4788 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:21.636921 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:21.631648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38683 (local address 127.14.212.126:39031)
1026 15:14:21.631858 (+ 210us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:21.631870 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:21.631893 (+ 23us) server_negotiation.cc:408] Connection header received
1026 15:14:21.632331 (+ 438us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:21.632340 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:21.632402 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:21.632548 (+ 146us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:21.632947 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:21.634109 (+ 1162us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:21.634885 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:21.635546 (+ 661us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:21.635877 (+ 331us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:21.635885 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:21.635885 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:21.636130 (+ 245us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:21.636154 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:21.636164 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:21.636551 (+ 387us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:21.636604 (+ 53us) server_negotiation.cc:300] Negotiation successful
1026 15:14:21.636630 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20251026 15:14:21.637971 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:21.638711 4788 catalog_manager.cc:1357] Generated new cluster ID: 8201753c2ad74d15b9ccd89ba2cd3f55
I20251026 15:14:21.638813 4788 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:21.642644 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:21.639525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46445 (local address 127.14.212.126:39031)
1026 15:14:21.639564 (+ 39us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:21.639566 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:21.639694 (+ 128us) server_negotiation.cc:408] Connection header received
1026 15:14:21.639788 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:21.639788 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:21.639797 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:21.639848 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:21.640214 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:21.640877 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:21.641617 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:21.641793 (+ 176us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:21.642357 (+ 564us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:21.642360 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:21.642361 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:21.642365 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:21.642377 (+ 12us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:21.642381 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:21.642474 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:21.642558 (+ 84us) server_negotiation.cc:300] Negotiation successful
1026 15:14:21.642568 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251026 15:14:21.649159 4788 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:21.649685 4788 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:21.653398 4788 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 30d46813f6554104a3bf21acf600b29d: Generated new TSK 0
I20251026 15:14:21.653565 4788 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:21.655848 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 4708
W20251026 15:14:21.660198 4690 connection.cc:537] client connection to 127.14.212.126:39031 recv error: Network error: failed to read from TLS socket (remote: 127.14.212.126:39031): Connection reset by peer (error 104)
2025-10-26T15:14:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:21 dist-test-slave-8pgz krb5kdc[4807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:21 dist-test-slave-8pgz krb5kdc[4807](info): set up 2 sockets
Oct 26 15:14:21 dist-test-slave-8pgz krb5kdc[4807](info): commencing operation
krb5kdc: starting...
W20251026 15:14:23.664153 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.990s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:23 dist-test-slave-8pgz krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:23.806758 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46009
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:46009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:23.888911 4823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:23.889132 4823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:23.889156 4823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:23.890728 4823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:23.890799 4823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:23.890828 4823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:23.890849 4823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:23.890892 4823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:23.892661 4823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:46009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:46009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:23.892889 4823 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:23.893056 4823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:23.895696 4829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:23.895660 4828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:23.895769 4823 server_base.cc:1047] running on GCE node
W20251026 15:14:23.895671 4831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:23.896243 4823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:23.896479 4823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:23.897640 4823 hybrid_clock.cc:648] HybridClock initialized: now 1761491663897624 us; error 36 us; skew 500 ppm
Oct 26 15:14:23 dist-test-slave-8pgz krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491663, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:23.900252 4823 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:14:23.900844 4823 webserver.cc:492] Webserver started at http://127.14.212.126:38843/ using document root <none> and password file <none>
I20251026 15:14:23.901053 4823 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:23.901091 4823 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:23.901202 4823 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:23.902158 4823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "3832dfe0347a433b861d4773e64691dc"
format_stamp: "Formatted at 2025-10-26 15:14:23 on dist-test-slave-8pgz"
server_key: "25e7fdf8427fdcec280cc647599d03aa"
server_key_iv: "22eac58819c1574e0d1c7e979721ca02"
server_key_version: "encryptionkey@0"
I20251026 15:14:23.902498 4823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "3832dfe0347a433b861d4773e64691dc"
format_stamp: "Formatted at 2025-10-26 15:14:23 on dist-test-slave-8pgz"
server_key: "25e7fdf8427fdcec280cc647599d03aa"
server_key_iv: "22eac58819c1574e0d1c7e979721ca02"
server_key_version: "encryptionkey@0"
I20251026 15:14:23.903721 4823 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:14:23.904453 4838 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:23.904650 4823 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 15:14:23.904726 4823 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "3832dfe0347a433b861d4773e64691dc"
format_stamp: "Formatted at 2025-10-26 15:14:23 on dist-test-slave-8pgz"
server_key: "25e7fdf8427fdcec280cc647599d03aa"
server_key_iv: "22eac58819c1574e0d1c7e979721ca02"
server_key_version: "encryptionkey@0"
I20251026 15:14:23.904820 4823 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:23.908228 4823 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:23.913866 4823 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:23.914099 4823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:23.918172 4823 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:46009
I20251026 15:14:23.918202 4890 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:46009 every 8 connection(s)
I20251026 15:14:23.918552 4823 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:23.919190 4891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:23.921416 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 4823
I20251026 15:14:23.921525 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:23.921686 15185 external_mini_cluster.cc:1468] Setting key 0fcdd7d26855f6c60226ec6d73b72980
I20251026 15:14:23.921706 4891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc: Bootstrap starting.
I20251026 15:14:23.922530 4891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:23.922848 4891 log.cc:826] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:23.923955 4891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc: No bootstrap required, opened a new log
I20251026 15:14:23.924147 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:23.922236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45303 (local address 127.14.212.126:46009)
1026 15:14:23.922446 (+ 210us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:23.922458 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:23.922486 (+ 28us) server_negotiation.cc:408] Connection header received
1026 15:14:23.922919 (+ 433us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:23.922928 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:23.922962 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:23.923111 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:23.923796 (+ 685us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:23.923800 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:23.923800 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:23.923805 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:23.923831 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:23.923838 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:23.923918 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:23.923957 (+ 39us) server_negotiation.cc:300] Negotiation successful
1026 15:14:23.923993 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20251026 15:14:23.925045 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:23.925380 4891 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3832dfe0347a433b861d4773e64691dc" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46009 } }
I20251026 15:14:23.925499 4891 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:23.925520 4891 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3832dfe0347a433b861d4773e64691dc, State: Initialized, Role: FOLLOWER
I20251026 15:14:23.925612 4891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3832dfe0347a433b861d4773e64691dc" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46009 } }
I20251026 15:14:23.925663 4891 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:23.925686 4891 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:23.925730 4891 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:23.926415 4891 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3832dfe0347a433b861d4773e64691dc" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46009 } }
I20251026 15:14:23.926544 4891 leader_election.cc:304] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3832dfe0347a433b861d4773e64691dc; no voters:
I20251026 15:14:23.926688 4891 leader_election.cc:290] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:23.926723 4901 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:23.926927 4901 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [term 1 LEADER]: Becoming Leader. State: Replica: 3832dfe0347a433b861d4773e64691dc, State: Running, Role: LEADER
I20251026 15:14:23.926998 4891 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:23.927055 4901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3832dfe0347a433b861d4773e64691dc" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46009 } }
I20251026 15:14:23.927374 4901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3832dfe0347a433b861d4773e64691dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3832dfe0347a433b861d4773e64691dc" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46009 } } }
I20251026 15:14:23.927455 4901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:23.927609 4901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3832dfe0347a433b861d4773e64691dc. Latest consensus state: current_term: 1 leader_uuid: "3832dfe0347a433b861d4773e64691dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3832dfe0347a433b861d4773e64691dc" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 46009 } } }
I20251026 15:14:23.927680 4906 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:23.927847 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:23.926509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52333 (local address 127.14.212.126:46009)
1026 15:14:23.926556 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:23.926558 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:23.926676 (+ 118us) server_negotiation.cc:408] Connection header received
1026 15:14:23.926755 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:23.926756 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:23.926762 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:23.926886 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:23.927564 (+ 678us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:23.927565 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:23.927566 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:23.927570 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:23.927586 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:23.927590 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:23.927647 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:23.927752 (+ 105us) server_negotiation.cc:300] Negotiation successful
1026 15:14:23.927768 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251026 15:14:23.927758 4901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:23.929028 4906 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:23.930691 4906 catalog_manager.cc:1357] Generated new cluster ID: ddc5fdb2948b496c85fef966d079120f
I20251026 15:14:23.930737 4906 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:23.941237 4906 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:23.941512 4906 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:23.950232 4906 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3832dfe0347a433b861d4773e64691dc: Generated new TSK 0
I20251026 15:14:23.950493 4906 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:23.961155 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 4823
2025-10-26T15:14:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:24 dist-test-slave-8pgz krb5kdc[4922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:24 dist-test-slave-8pgz krb5kdc[4922](info): set up 2 sockets
Oct 26 15:14:24 dist-test-slave-8pgz krb5kdc[4922](info): commencing operation
krb5kdc: starting...
W20251026 15:14:25.988423 15185 mini_kdc.cc:121] Time spent starting KDC: real 2.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:26 dist-test-slave-8pgz krb5kdc[4922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:26.129424 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:43733
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:43733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:26.212157 4938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:26.212375 4938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:26.212416 4938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:26.213940 4938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:26.213995 4938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:26.214036 4938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:26.214061 4938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:26.214087 4938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:26.215865 4938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:43733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:43733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:26.216148 4938 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:26.216346 4938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:26.218936 4944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:26.218925 4943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:26.219103 4938 server_base.cc:1047] running on GCE node
W20251026 15:14:26.218925 4946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:26.219379 4938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:26.219612 4938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:26.220762 4938 hybrid_clock.cc:648] HybridClock initialized: now 1761491666220728 us; error 49 us; skew 500 ppm
Oct 26 15:14:26 dist-test-slave-8pgz krb5kdc[4922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491666, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:26.223474 4938 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:14:26.224011 4938 webserver.cc:492] Webserver started at http://127.14.212.126:37023/ using document root <none> and password file <none>
I20251026 15:14:26.224273 4938 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:26.224325 4938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:26.224444 4938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:26.225392 4938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "cbbdd0ca52fd48c199e9362f8548ad42"
format_stamp: "Formatted at 2025-10-26 15:14:26 on dist-test-slave-8pgz"
server_key: "4c3a1e93759d20d996081c6dbe6cabca"
server_key_iv: "b0ec9c34f6da6c35174bbb1740302e05"
server_key_version: "encryptionkey@0"
I20251026 15:14:26.225689 4938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "cbbdd0ca52fd48c199e9362f8548ad42"
format_stamp: "Formatted at 2025-10-26 15:14:26 on dist-test-slave-8pgz"
server_key: "4c3a1e93759d20d996081c6dbe6cabca"
server_key_iv: "b0ec9c34f6da6c35174bbb1740302e05"
server_key_version: "encryptionkey@0"
I20251026 15:14:26.226998 4938 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 15:14:26.227757 4953 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:26.227972 4938 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:14:26.228051 4938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "cbbdd0ca52fd48c199e9362f8548ad42"
format_stamp: "Formatted at 2025-10-26 15:14:26 on dist-test-slave-8pgz"
server_key: "4c3a1e93759d20d996081c6dbe6cabca"
server_key_iv: "b0ec9c34f6da6c35174bbb1740302e05"
server_key_version: "encryptionkey@0"
I20251026 15:14:26.228127 4938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:26.244632 4938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:26.246819 4938 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:26.246961 4938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:26.250792 4938 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:43733
I20251026 15:14:26.250838 5005 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:43733 every 8 connection(s)
I20251026 15:14:26.251295 4938 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:26.251932 5006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:26.253960 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 4938
I20251026 15:14:26.254061 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:26.254227 15185 external_mini_cluster.cc:1468] Setting key 661034b95fb70af3bc223647944681e0
I20251026 15:14:26.254545 5006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42: Bootstrap starting.
I20251026 15:14:26.255403 5006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:26.255694 5006 log.cc:826] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:26.256435 5006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42: No bootstrap required, opened a new log
I20251026 15:14:26.258083 5006 raft_consensus.cc:359] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbdd0ca52fd48c199e9362f8548ad42" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43733 } }
I20251026 15:14:26.258234 5006 raft_consensus.cc:385] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:26.258263 5006 raft_consensus.cc:740] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbbdd0ca52fd48c199e9362f8548ad42, State: Initialized, Role: FOLLOWER
I20251026 15:14:26.258368 5006 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbdd0ca52fd48c199e9362f8548ad42" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43733 } }
I20251026 15:14:26.258472 5006 raft_consensus.cc:399] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:26.258503 5006 raft_consensus.cc:493] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:26.258561 5006 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:26.259400 5006 raft_consensus.cc:515] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbdd0ca52fd48c199e9362f8548ad42" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43733 } }
I20251026 15:14:26.259518 5006 leader_election.cc:304] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbbdd0ca52fd48c199e9362f8548ad42; no voters:
I20251026 15:14:26.259696 5006 leader_election.cc:290] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:26.259745 5011 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:26.259894 5011 raft_consensus.cc:697] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [term 1 LEADER]: Becoming Leader. State: Replica: cbbdd0ca52fd48c199e9362f8548ad42, State: Running, Role: LEADER
I20251026 15:14:26.260030 5006 sys_catalog.cc:565] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:26.260025 5011 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbdd0ca52fd48c199e9362f8548ad42" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43733 } }
I20251026 15:14:26.260406 5013 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbbdd0ca52fd48c199e9362f8548ad42. Latest consensus state: current_term: 1 leader_uuid: "cbbdd0ca52fd48c199e9362f8548ad42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbdd0ca52fd48c199e9362f8548ad42" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43733 } } }
I20251026 15:14:26.260496 5013 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:26.260396 5012 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbbdd0ca52fd48c199e9362f8548ad42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbdd0ca52fd48c199e9362f8548ad42" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 43733 } } }
I20251026 15:14:26.260790 5012 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42 [sys.catalog]: This master's current role is: LEADER
Oct 26 15:14:26 dist-test-slave-8pgz krb5kdc[4922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:14:26.261057 5015 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:26.261535 5015 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:26.263536 5015 catalog_manager.cc:1357] Generated new cluster ID: 3b33cacd29274b6daf79554d2f949fa9
I20251026 15:14:26.263590 5015 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:26.265166 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:26.254760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44137 (local address 127.14.212.126:43733)
1026 15:14:26.255032 (+ 272us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:26.255047 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:26.255076 (+ 29us) server_negotiation.cc:408] Connection header received
1026 15:14:26.255560 (+ 484us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:26.255571 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:26.255633 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:26.255785 (+ 152us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:26.256653 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:26.257835 (+ 1182us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:26.258622 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:26.258938 (+ 316us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:26.261427 (+ 2489us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:26.261436 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:26.261437 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:26.261462 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:26.263886 (+ 2424us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:26.264299 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:26.264302 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:26.264302 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:26.264356 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:26.264501 (+ 145us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:26.264501 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:26.264501 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:26.264684 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:26.264802 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:26.264884 (+ 82us) server_negotiation.cc:300] Negotiation successful
1026 15:14:26.264980 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":90,"threads_started":1}
I20251026 15:14:26.266039 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:26.274237 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:26.267505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38117 (local address 127.14.212.126:43733)
1026 15:14:26.267561 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:26.267563 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:26.267677 (+ 114us) server_negotiation.cc:408] Connection header received
1026 15:14:26.267741 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:26.267742 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:26.267753 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:26.267833 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:26.268586 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:26.269298 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:26.269991 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:26.270176 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:26.271443 (+ 1267us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:26.271448 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:26.271448 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:26.271456 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:26.273192 (+ 1736us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:26.273522 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:26.273523 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:26.273523 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:26.273546 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:26.273699 (+ 153us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:26.273700 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:26.273700 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:26.273820 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:26.273916 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:26.274032 (+ 116us) server_negotiation.cc:300] Negotiation successful
1026 15:14:26.274123 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251026 15:14:26.289860 5015 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:26.290621 5015 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:26.301290 5015 catalog_manager.cc:6022] T 00000000000000000000000000000000 P cbbdd0ca52fd48c199e9362f8548ad42: Generated new TSK 0
I20251026 15:14:26.301496 5015 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:26.308466 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 4938
2025-10-26T15:14:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:26 dist-test-slave-8pgz krb5kdc[5037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:26 dist-test-slave-8pgz krb5kdc[5037](info): set up 2 sockets
Oct 26 15:14:26 dist-test-slave-8pgz krb5kdc[5037](info): commencing operation
krb5kdc: starting...
W20251026 15:14:28.292249 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.965s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:28 dist-test-slave-8pgz krb5kdc[5037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:28.433816 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:44793
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:44793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:28.516366 5053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:28.516583 5053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:28.516623 5053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:28.518151 5053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:28.518210 5053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:28.518230 5053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:28.518251 5053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:28.518268 5053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:28.520056 5053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:44793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:44793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:28.520340 5053 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:28.520535 5053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:28.523204 5059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:28.523313 5053 server_base.cc:1047] running on GCE node
W20251026 15:14:28.523290 5058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:28.523204 5061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:28.523689 5053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:28.523940 5053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:28.525112 5053 hybrid_clock.cc:648] HybridClock initialized: now 1761491668525086 us; error 46 us; skew 500 ppm
Oct 26 15:14:28 dist-test-slave-8pgz krb5kdc[5037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491668, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:28.527740 5053 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:14:28.528326 5053 webserver.cc:492] Webserver started at http://127.14.212.126:40281/ using document root <none> and password file <none>
I20251026 15:14:28.528517 5053 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:28.528556 5053 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:28.528673 5053 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:28.529637 5053 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "3861c0077c2645ab82b4a89c679017f3"
format_stamp: "Formatted at 2025-10-26 15:14:28 on dist-test-slave-8pgz"
server_key: "c7e9f77069b869cd21abbfc54c6fb046"
server_key_iv: "62fbd6851ae4ba10ca020fd0d3b1be01"
server_key_version: "encryptionkey@0"
I20251026 15:14:28.529938 5053 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "3861c0077c2645ab82b4a89c679017f3"
format_stamp: "Formatted at 2025-10-26 15:14:28 on dist-test-slave-8pgz"
server_key: "c7e9f77069b869cd21abbfc54c6fb046"
server_key_iv: "62fbd6851ae4ba10ca020fd0d3b1be01"
server_key_version: "encryptionkey@0"
I20251026 15:14:28.531198 5053 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 15:14:28.531883 5068 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:28.532042 5053 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:28.532121 5053 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "3861c0077c2645ab82b4a89c679017f3"
format_stamp: "Formatted at 2025-10-26 15:14:28 on dist-test-slave-8pgz"
server_key: "c7e9f77069b869cd21abbfc54c6fb046"
server_key_iv: "62fbd6851ae4ba10ca020fd0d3b1be01"
server_key_version: "encryptionkey@0"
I20251026 15:14:28.532181 5053 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:28.551846 5053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:28.553892 5053 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:28.554029 5053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:28.558130 5053 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:44793
I20251026 15:14:28.558185 5120 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:44793 every 8 connection(s)
I20251026 15:14:28.558552 5053 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:28.559259 5121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:28.561725 5121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3: Bootstrap starting.
I20251026 15:14:28.562314 5121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:28.562582 5121 log.cc:826] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:28.563364 5121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3: No bootstrap required, opened a new log
I20251026 15:14:28.564618 5121 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3861c0077c2645ab82b4a89c679017f3" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44793 } }
I20251026 15:14:28.564746 5121 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:28.564766 5121 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3861c0077c2645ab82b4a89c679017f3, State: Initialized, Role: FOLLOWER
I20251026 15:14:28.564884 5121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3861c0077c2645ab82b4a89c679017f3" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44793 } }
I20251026 15:14:28.564965 5121 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:28.565007 5121 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:28.565047 5121 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:28.565575 5121 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3861c0077c2645ab82b4a89c679017f3" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44793 } }
I20251026 15:14:28.565670 5121 leader_election.cc:304] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3861c0077c2645ab82b4a89c679017f3; no voters:
I20251026 15:14:28.565815 5121 leader_election.cc:290] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:28.565896 5124 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:28.566066 5124 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [term 1 LEADER]: Becoming Leader. State: Replica: 3861c0077c2645ab82b4a89c679017f3, State: Running, Role: LEADER
I20251026 15:14:28.566138 5121 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:28.566164 5124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3861c0077c2645ab82b4a89c679017f3" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44793 } }
I20251026 15:14:28.566571 5126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3861c0077c2645ab82b4a89c679017f3. Latest consensus state: current_term: 1 leader_uuid: "3861c0077c2645ab82b4a89c679017f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3861c0077c2645ab82b4a89c679017f3" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44793 } } }
I20251026 15:14:28.566607 5125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3861c0077c2645ab82b4a89c679017f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3861c0077c2645ab82b4a89c679017f3" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 44793 } } }
I20251026 15:14:28.566658 5126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:28.566679 5125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:28.566848 5130 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:28.567307 5130 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:28.568653 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 5053
I20251026 15:14:28.568768 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:28.568933 5130 catalog_manager.cc:1357] Generated new cluster ID: 876356a087c74d16a5d474d55d122dcd
I20251026 15:14:28.568949 15185 external_mini_cluster.cc:1468] Setting key edc3dd5a439243e70b8195ef66459a6c
I20251026 15:14:28.568991 5130 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:28.573192 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:28.569410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56699 (local address 127.14.212.126:44793)
1026 15:14:28.569576 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:28.569588 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:28.569613 (+ 25us) server_negotiation.cc:408] Connection header received
1026 15:14:28.569753 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:28.569757 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:28.569797 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:28.569904 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:28.570308 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:28.571103 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:28.571691 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:28.572431 (+ 740us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:28.572531 (+ 100us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:28.572536 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:28.572536 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:28.572734 (+ 198us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:28.572747 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:28.572754 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:28.572825 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:28.572965 (+ 140us) server_negotiation.cc:300] Negotiation successful
1026 15:14:28.573010 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20251026 15:14:28.574026 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:28.577812 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:28.575392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53041 (local address 127.14.212.126:44793)
1026 15:14:28.575428 (+ 36us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:28.575430 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:28.575577 (+ 147us) server_negotiation.cc:408] Connection header received
1026 15:14:28.575639 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:28.575640 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:28.575651 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:28.575706 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:28.575952 (+ 246us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:28.576407 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:28.576920 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:28.577445 (+ 525us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:28.577542 (+ 97us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:28.577546 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:28.577546 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:28.577550 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:28.577561 (+ 11us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:28.577564 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:28.577625 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:28.577694 (+ 69us) server_negotiation.cc:300] Negotiation successful
1026 15:14:28.577721 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20251026 15:14:28.588449 5130 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:28.589000 5130 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:28.594362 5130 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3861c0077c2645ab82b4a89c679017f3: Generated new TSK 0
I20251026 15:14:28.594583 5130 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:28.604012 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 5053
2025-10-26T15:14:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:28 dist-test-slave-8pgz krb5kdc[5152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:28 dist-test-slave-8pgz krb5kdc[5152](info): set up 2 sockets
Oct 26 15:14:28 dist-test-slave-8pgz krb5kdc[5152](info): commencing operation
krb5kdc: starting...
W20251026 15:14:30.644119 15185 mini_kdc.cc:121] Time spent starting KDC: real 2.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:30 dist-test-slave-8pgz krb5kdc[5152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:30.794113 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33827
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:42737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:33827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:30.874668 5168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:30.874887 5168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:30.874913 5168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:30.876418 5168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:30.876473 5168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:30.876488 5168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:30.876500 5168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:30.876513 5168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:30.878208 5168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:42737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:33827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:30.878495 5168 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:30.878707 5168 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:30.881397 5173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:30.881404 5176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:30.881578 5174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:30.881703 5168 server_base.cc:1047] running on GCE node
I20251026 15:14:30.881953 5168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:30.882176 5168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:30.883360 5168 hybrid_clock.cc:648] HybridClock initialized: now 1761491670883346 us; error 31 us; skew 500 ppm
Oct 26 15:14:30 dist-test-slave-8pgz krb5kdc[5152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491670, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:30.886252 5168 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:14:30.886870 5168 webserver.cc:492] Webserver started at http://127.14.212.126:36215/ using document root <none> and password file <none>
I20251026 15:14:30.887082 5168 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:30.887130 5168 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:30.887236 5168 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:30.888213 5168 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "a408d7a30ac54f459fc4d8c6759e1928"
format_stamp: "Formatted at 2025-10-26 15:14:30 on dist-test-slave-8pgz"
server_key: "a7ef5b54cf93a73f1b9157187a8f608a"
server_key_iv: "a2f28e58c8016014710132fe8d6cd377"
server_key_version: "encryptionkey@0"
I20251026 15:14:30.888548 5168 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "a408d7a30ac54f459fc4d8c6759e1928"
format_stamp: "Formatted at 2025-10-26 15:14:30 on dist-test-slave-8pgz"
server_key: "a7ef5b54cf93a73f1b9157187a8f608a"
server_key_iv: "a2f28e58c8016014710132fe8d6cd377"
server_key_version: "encryptionkey@0"
I20251026 15:14:30.889830 5168 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:14:30.890565 5183 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:30.890717 5168 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:14:30.890822 5168 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "a408d7a30ac54f459fc4d8c6759e1928"
format_stamp: "Formatted at 2025-10-26 15:14:30 on dist-test-slave-8pgz"
server_key: "a7ef5b54cf93a73f1b9157187a8f608a"
server_key_iv: "a2f28e58c8016014710132fe8d6cd377"
server_key_version: "encryptionkey@0"
I20251026 15:14:30.890880 5168 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:30.894188 5168 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:30.901198 5168 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:30.901340 5168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:30.905551 5168 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:33827
I20251026 15:14:30.905584 5235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:33827 every 8 connection(s)
I20251026 15:14:30.905915 5168 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:30.906562 5236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:30.908950 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 5168
I20251026 15:14:30.909046 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:30.909073 5236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928: Bootstrap starting.
I20251026 15:14:30.909219 15185 external_mini_cluster.cc:1468] Setting key 8dc5717ee5b98d1531bb7d3250a54aa0
I20251026 15:14:30.909750 5236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:30.910061 5236 log.cc:826] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:30.910904 5236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928: No bootstrap required, opened a new log
I20251026 15:14:30.911623 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:30.909759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58423 (local address 127.14.212.126:33827)
1026 15:14:30.909942 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:30.909952 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:30.909972 (+ 20us) server_negotiation.cc:408] Connection header received
1026 15:14:30.910327 (+ 355us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:30.910333 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:30.910362 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:30.910519 (+ 157us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:30.911241 (+ 722us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:30.911245 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:30.911245 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:30.911249 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:30.911272 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:30.911278 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:30.911343 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:30.911416 (+ 73us) server_negotiation.cc:300] Negotiation successful
1026 15:14:30.911447 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20251026 15:14:30.912415 5236 raft_consensus.cc:359] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a408d7a30ac54f459fc4d8c6759e1928" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33827 } }
I20251026 15:14:30.912575 5236 raft_consensus.cc:385] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 15:14:30.912608 5236 raft_consensus.cc:740] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a408d7a30ac54f459fc4d8c6759e1928, State: Initialized, Role: FOLLOWER
I20251026 15:14:30.912726 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:30.912714 5236 consensus_queue.cc:260] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a408d7a30ac54f459fc4d8c6759e1928" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33827 } }
I20251026 15:14:30.912777 5236 raft_consensus.cc:399] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 15:14:30.912822 5236 raft_consensus.cc:493] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 15:14:30.912868 5236 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:30.913653 5236 raft_consensus.cc:515] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a408d7a30ac54f459fc4d8c6759e1928" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33827 } }
I20251026 15:14:30.913784 5236 leader_election.cc:304] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a408d7a30ac54f459fc4d8c6759e1928; no voters:
I20251026 15:14:30.913966 5236 leader_election.cc:290] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 15:14:30.914073 5244 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:30.914212 5236 sys_catalog.cc:565] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:30.914275 5244 raft_consensus.cc:697] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [term 1 LEADER]: Becoming Leader. State: Replica: a408d7a30ac54f459fc4d8c6759e1928, State: Running, Role: LEADER
I20251026 15:14:30.914373 5244 consensus_queue.cc:237] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a408d7a30ac54f459fc4d8c6759e1928" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33827 } }
I20251026 15:14:30.914739 5246 sys_catalog.cc:455] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a408d7a30ac54f459fc4d8c6759e1928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a408d7a30ac54f459fc4d8c6759e1928" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33827 } } }
I20251026 15:14:30.914844 5246 sys_catalog.cc:458] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:30.914738 5247 sys_catalog.cc:455] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a408d7a30ac54f459fc4d8c6759e1928. Latest consensus state: current_term: 1 leader_uuid: "a408d7a30ac54f459fc4d8c6759e1928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a408d7a30ac54f459fc4d8c6759e1928" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33827 } } }
I20251026 15:14:30.915023 5247 sys_catalog.cc:458] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:30.915545 5254 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:30.916174 5254 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:30.916683 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:30.914357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48621 (local address 127.14.212.126:33827)
1026 15:14:30.914424 (+ 67us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:30.914426 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:30.915317 (+ 891us) server_negotiation.cc:408] Connection header received
1026 15:14:30.915467 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:30.915468 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:30.915474 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:30.915543 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:30.916312 (+ 769us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:30.916313 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:30.916314 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1026 15:14:30.916318 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:30.916334 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1026 15:14:30.916338 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:30.916399 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:30.916605 (+ 206us) server_negotiation.cc:300] Negotiation successful
1026 15:14:30.916619 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20251026 15:14:30.917999 5254 catalog_manager.cc:1357] Generated new cluster ID: f4aa2c3efc604feb9b5132a581184016
I20251026 15:14:30.918051 5254 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:30.929709 5254 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:30.929912 5254 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:30.935266 5254 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a408d7a30ac54f459fc4d8c6759e1928: Generated new TSK 0
I20251026 15:14:30.935464 5254 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 15:14:30.943524 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 5168
2025-10-26T15:14:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2339 ms)
[----------] 18 tests from AuthTokenIssuingTest (41543 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:31 dist-test-slave-8pgz krb5kdc[5267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:31 dist-test-slave-8pgz krb5kdc[5267](info): set up 2 sockets
Oct 26 15:14:31 dist-test-slave-8pgz krb5kdc[5267](info): commencing operation
krb5kdc: starting...
W20251026 15:14:32.944586 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.983s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:33.089922 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33363
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:33363,127.14.212.125:39319,127.14.212.124:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:33.171178 5283 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251026 15:14:33.171401 5283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:33.171430 5283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:33.171464 5283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:33.172974 5283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:33.173029 5283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:33.173050 5283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:33.173069 5283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:33.173090 5283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:33.174839 5283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:33363,127.14.212.125:39319,127.14.212.124:33723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:33.175136 5283 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:33.175328 5283 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:33.178087 5289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:33.178145 5283 server_base.cc:1047] running on GCE node
W20251026 15:14:33.178093 5291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:33.178093 5288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:33.178571 5283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:33.178797 5283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:33.179979 5283 hybrid_clock.cc:648] HybridClock initialized: now 1761491673179964 us; error 27 us; skew 500 ppm
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:33.182642 5283 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:14:33.183213 5283 webserver.cc:492] Webserver started at http://127.14.212.126:45923/ using document root <none> and password file <none>
I20251026 15:14:33.183394 5283 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:33.183427 5283 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:33.183508 5283 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:33.184445 5283 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2"
format_stamp: "Formatted at 2025-10-26 15:14:33 on dist-test-slave-8pgz"
server_key: "4333a9ffa2905a3c5577f05163d5ce33"
server_key_iv: "a08b4b57e70dd7cc0288736a3b0988a0"
server_key_version: "encryptionkey@0"
I20251026 15:14:33.184721 5283 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2"
format_stamp: "Formatted at 2025-10-26 15:14:33 on dist-test-slave-8pgz"
server_key: "4333a9ffa2905a3c5577f05163d5ce33"
server_key_iv: "a08b4b57e70dd7cc0288736a3b0988a0"
server_key_version: "encryptionkey@0"
I20251026 15:14:33.185961 5283 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 15:14:33.186805 5298 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:33.187036 5283 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:14:33.187114 5283 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2"
format_stamp: "Formatted at 2025-10-26 15:14:33 on dist-test-slave-8pgz"
server_key: "4333a9ffa2905a3c5577f05163d5ce33"
server_key_iv: "a08b4b57e70dd7cc0288736a3b0988a0"
server_key_version: "encryptionkey@0"
I20251026 15:14:33.187172 5283 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:33.210191 5283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:33.212389 5283 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:33.212505 5283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:33.216406 5283 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:33363
I20251026 15:14:33.216449 5350 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:33363 every 8 connection(s)
I20251026 15:14:33.216777 5283 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:33.217406 5351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:33.219324 5351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:14:33.224855 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 5283
I20251026 15:14:33.224987 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:33.225188 15185 external_mini_cluster.cc:1468] Setting key 691983d588ba70167f5dda7b49ffe419
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:14:33.229171 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.220043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.14.212.126:33363)
1026 15:14:33.220242 (+ 199us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.220249 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.220273 (+ 24us) server_negotiation.cc:408] Connection header received
1026 15:14:33.220680 (+ 407us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.220682 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.220718 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.220815 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.221719 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.222837 (+ 1118us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.223400 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.223583 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.226027 (+ 2444us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.226035 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.226035 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.226042 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.227815 (+ 1773us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.228189 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.228192 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.228192 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.228231 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.228400 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.228401 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.228401 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.228584 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.228665 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.228832 (+ 167us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.228963 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":103,"threads_started":1}
I20251026 15:14:33.229179 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.219971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33363 (local address 127.0.0.1:46134)
1026 15:14:33.220130 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.220163 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:33.220518 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:33.220841 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:33.220844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:33.221284 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:33.221602 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.221606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.222932 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.222933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:33.223305 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.223306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.223434 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.224075 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:33.224103 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:33.225918 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:33.227926 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.227932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.227935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.228107 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.228298 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.228299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.228299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.228342 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.228648 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:33.228652 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:33.228718 (+ 66us) client_negotiation.cc:769] Sending connection context
1026 15:14:33.228799 (+ 81us) client_negotiation.cc:240] Negotiation successful
1026 15:14:33.229044 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":123,"thread_start_us":67,"threads_started":1}
I20251026 15:14:33.229909 5351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:14:33.230358 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.230207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:39319 (local address 127.0.0.1:43094)
1026 15:14:33.230273 (+ 66us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.230325 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20251026 15:14:33.230509 5299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.125:39319: connect: Connection refused (error 111)
I20251026 15:14:33.230988 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.230862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:39319 (local address 127.0.0.1:43110)
1026 15:14:33.230925 (+ 63us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.230959 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20251026 15:14:33.231117 5351 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.125:39319: Network error: Client connection negotiation failed: client connection to 127.14.212.125:39319: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.14.212.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:33.277393 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.125:39319
--webserver_interface=127.14.212.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:33363,127.14.212.125:39319,127.14.212.124:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:14:33.280319 5351 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } attempt: 1
I20251026 15:14:33.280885 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.280696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:39319 (local address 127.0.0.1:43124)
1026 15:14:33.280809 (+ 113us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.280851 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20251026 15:14:33.281363 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.281239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:39319 (local address 127.0.0.1:43138)
1026 15:14:33.281310 (+ 71us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.281339 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20251026 15:14:33.281476 5351 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.125:39319: Network error: Client connection negotiation failed: client connection to 127.14.212.125:39319: connect: Connection refused (error 111)
I20251026 15:14:33.349670 5351 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } attempt: 2
I20251026 15:14:33.350267 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.350131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:39319 (local address 127.0.0.1:43140)
1026 15:14:33.350194 (+ 63us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.350237 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20251026 15:14:33.350661 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.350565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:39319 (local address 127.0.0.1:43148)
1026 15:14:33.350616 (+ 51us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.350642 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:39319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20251026 15:14:33.350780 5351 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.125:39319: Network error: Client connection negotiation failed: client connection to 127.14.212.125:39319: connect: Connection refused (error 111)
W20251026 15:14:33.359373 5360 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251026 15:14:33.359591 5360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:33.359608 5360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:33.359632 5360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:33.361095 5360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:33.361145 5360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:33.361160 5360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:33.361176 5360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:33.361187 5360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:33.363080 5360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:33363,127.14.212.125:39319,127.14.212.124:33723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.125:39319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.212.125
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:33.363301 5360 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:33.363461 5360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:33.366307 5368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:33.366297 5366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:33.366298 5365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:33.366330 5360 server_base.cc:1047] running on GCE node
I20251026 15:14:33.366722 5360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:33.367020 5360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:33.368191 5360 hybrid_clock.cc:648] HybridClock initialized: now 1761491673368166 us; error 37 us; skew 500 ppm
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:33.370683 5360 init.cc:377] Logged in from keytab as kudu/127.14.212.125@KRBTEST.COM (short username kudu)
I20251026 15:14:33.371250 5360 webserver.cc:492] Webserver started at http://127.14.212.125:37063/ using document root <none> and password file <none>
I20251026 15:14:33.371438 5360 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:33.371477 5360 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:33.371595 5360 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:33.372555 5360 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/data/instance:
uuid: "e18471498f4a4b4ca0ec50589ead2069"
format_stamp: "Formatted at 2025-10-26 15:14:33 on dist-test-slave-8pgz"
server_key: "1d51da89e837816bbf30993d636dadef"
server_key_iv: "78164cb8a76a1003566b167acfed19a3"
server_key_version: "encryptionkey@0"
I20251026 15:14:33.372875 5360 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/wal/instance:
uuid: "e18471498f4a4b4ca0ec50589ead2069"
format_stamp: "Formatted at 2025-10-26 15:14:33 on dist-test-slave-8pgz"
server_key: "1d51da89e837816bbf30993d636dadef"
server_key_iv: "78164cb8a76a1003566b167acfed19a3"
server_key_version: "encryptionkey@0"
I20251026 15:14:33.374049 5360 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:14:33.374779 5375 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:33.374953 5360 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:33.375025 5360 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/wal
uuid: "e18471498f4a4b4ca0ec50589ead2069"
format_stamp: "Formatted at 2025-10-26 15:14:33 on dist-test-slave-8pgz"
server_key: "1d51da89e837816bbf30993d636dadef"
server_key_iv: "78164cb8a76a1003566b167acfed19a3"
server_key_version: "encryptionkey@0"
I20251026 15:14:33.375087 5360 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:33.399513 5360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:33.401386 5360 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:33.401522 5360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:33.405824 5360 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.125:39319
I20251026 15:14:33.405872 5427 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.125:39319 every 8 connection(s)
I20251026 15:14:33.406189 5360 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/data/info.pb
I20251026 15:14:33.406883 5428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:33.408725 5428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:14:33.412482 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 5360
I20251026 15:14:33.412618 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-1/wal/instance
I20251026 15:14:33.412799 15185 external_mini_cluster.cc:1468] Setting key 377bf0a3c21dab41951ab317494787c5
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.125@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:14:33.417516 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.409435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.14.212.126:33363)
1026 15:14:33.409504 (+ 69us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.409506 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.409656 (+ 150us) server_negotiation.cc:408] Connection header received
1026 15:14:33.410054 (+ 398us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.410055 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.410067 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.410135 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.410981 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.411439 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.412067 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.412215 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.414523 (+ 2308us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.414531 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.414532 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.414539 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.416197 (+ 1658us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.416619 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.416620 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.416620 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.416643 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.416868 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.416869 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.416870 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.416967 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.417063 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.417325 (+ 262us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.417421 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251026 15:14:33.417694 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.409375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33363 (local address 127.0.0.1:46136)
1026 15:14:33.409551 (+ 176us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.409579 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:33.409955 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:33.410231 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:33.410235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:33.410573 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:33.410862 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.410865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.411550 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.411551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:33.411944 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.411946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.412070 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.412707 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:33.412785 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:33.414383 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:33.416302 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.416310 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.416314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.416537 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.416706 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.416706 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.416707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.416801 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.417050 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:33.417056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:33.417196 (+ 140us) client_negotiation.cc:769] Sending connection context
1026 15:14:33.417294 (+ 98us) client_negotiation.cc:240] Negotiation successful
1026 15:14:33.417438 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20251026 15:14:33.418287 5428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.212.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.124@KRBTEST.COM" created.
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.125@KRBTEST.COM for kudu/127.14.212.125@KRBTEST.COM
I20251026 15:14:33.427078 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.418699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.14.212.125:39319)
1026 15:14:33.418856 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.418863 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.418874 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:14:33.418890 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.418891 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.418902 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.418961 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.419338 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.420034 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.420517 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.420718 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.423648 (+ 2930us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.423660 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.423661 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.423667 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.425636 (+ 1969us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.426115 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.426118 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.426118 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.426157 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.426374 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.426375 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.426375 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.426538 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.426623 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.426806 (+ 183us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.426959 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20251026 15:14:33.427085 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.418604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:39319 (local address 127.0.0.1:43160)
1026 15:14:33.418655 (+ 51us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.418665 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:33.418704 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:33.418983 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:33.418984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:33.419136 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:33.419263 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.419265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.420117 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.420118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:33.420444 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.420445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.420509 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.421114 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:33.421118 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:33.423515 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:33.425758 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.425759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.425760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.425971 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.426247 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.426248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.426248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.426285 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.426613 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:33.426614 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:33.426673 (+ 59us) client_negotiation.cc:769] Sending connection context
1026 15:14:33.426755 (+ 82us) client_negotiation.cc:240] Negotiation successful
1026 15:14:33.426885 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20251026 15:14:33.427789 5428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:14:33.428161 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.427994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50126)
1026 15:14:33.428074 (+ 80us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.428128 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20251026 15:14:33.428300 5379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
I20251026 15:14:33.428725 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.428609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50138)
1026 15:14:33.428662 (+ 53us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.428697 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20251026 15:14:33.428884 5428 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:33723: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:33.441944 5351 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } attempt: 3
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for kudu/127.14.212.125@KRBTEST.COM
WARNING: no policy specified for HTTP/127.14.212.124@KRBTEST.COM; defaulting to no policy
I20251026 15:14:33.448626 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.442422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43170 (local address 127.14.212.125:39319)
1026 15:14:33.442490 (+ 68us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.442492 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.442503 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:14:33.442586 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.442587 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.442603 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.442652 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.443102 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.443559 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.444077 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.444280 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.446125 (+ 1845us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.446129 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.446129 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.446136 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.447641 (+ 1505us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.447960 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.447961 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.447961 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.447981 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.448120 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.448120 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.448120 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.448206 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.448258 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.448483 (+ 225us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.448558 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251026 15:14:33.448633 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.442326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:39319 (local address 127.0.0.1:43170)
1026 15:14:33.442449 (+ 123us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.442462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:33.442519 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:33.442667 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:33.442668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:33.442864 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:33.443008 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.443010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.443641 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.443642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:33.443994 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.443995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.444060 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.444675 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:33.444679 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:33.446025 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:33.447747 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.447749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.447750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.447897 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.448051 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.448052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.448052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.448077 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.448298 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:33.448300 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:33.448371 (+ 71us) client_negotiation.cc:769] Sending connection context
1026 15:14:33.448457 (+ 86us) client_negotiation.cc:240] Negotiation successful
1026 15:14:33.448553 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.124@KRBTEST.COM" created.
I20251026 15:14:33.449100 5351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:14:33.449460 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.449334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50140)
1026 15:14:33.449395 (+ 61us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.449431 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20251026 15:14:33.449851 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.449726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50148)
1026 15:14:33.449810 (+ 84us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.449832 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20251026 15:14:33.449941 5351 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:33723: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:33.466287 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.124:33723
--webserver_interface=127.14.212.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:34533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:33363,127.14.212.125:39319,127.14.212.124:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:14:33.478137 5428 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } attempt: 1
I20251026 15:14:33.478657 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.478527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50152)
1026 15:14:33.478588 (+ 61us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.478628 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20251026 15:14:33.479094 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.478950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50158)
1026 15:14:33.479041 (+ 91us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.479072 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20251026 15:14:33.479197 5428 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:33723: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
I20251026 15:14:33.481110 5351 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } attempt: 1
I20251026 15:14:33.481540 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.481417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50162)
1026 15:14:33.481475 (+ 58us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.481511 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20251026 15:14:33.481904 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.481808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50166)
1026 15:14:33.481857 (+ 49us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.481882 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20251026 15:14:33.482019 5351 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:33723: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
W20251026 15:14:33.546736 5437 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251026 15:14:33.546949 5437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:33.546967 5437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:33.546988 5437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251026 15:14:33.547348 5428 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } attempt: 2
I20251026 15:14:33.547819 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.547680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50176)
1026 15:14:33.547753 (+ 73us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.547791 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20251026 15:14:33.548193 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.548101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50184)
1026 15:14:33.548150 (+ 49us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.548175 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20251026 15:14:33.548288 5428 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:33723: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
W20251026 15:14:33.548688 5437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:33.548734 5437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:33.548760 5437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:33.548797 5437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:33.548820 5437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:33.550496 5437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:34533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:33363,127.14.212.125:39319,127.14.212.124:33723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.124:33723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.212.124
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:33.550752 5437 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:33.550951 5437 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:33.553434 5443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:33.553472 5437 server_base.cc:1047] running on GCE node
W20251026 15:14:33.553536 5442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:33.553664 5445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:33.553994 5437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:33.554263 5437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:33.555400 5437 hybrid_clock.cc:648] HybridClock initialized: now 1761491673555380 us; error 35 us; skew 500 ppm
I20251026 15:14:33.557179 5351 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } attempt: 2
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:33.557626 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.557463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50186)
1026 15:14:33.557531 (+ 68us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.557583 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20251026 15:14:33.557991 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.557902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50188)
1026 15:14:33.557947 (+ 45us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.557971 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20251026 15:14:33.557984 5437 init.cc:377] Logged in from keytab as kudu/127.14.212.124@KRBTEST.COM (short username kudu)
W20251026 15:14:33.558091 5351 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:33723: Network error: Client connection negotiation failed: client connection to 127.14.212.124:33723: connect: Connection refused (error 111)
I20251026 15:14:33.558583 5437 webserver.cc:492] Webserver started at http://127.14.212.124:34323/ using document root <none> and password file <none>
I20251026 15:14:33.558799 5437 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:33.558840 5437 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:33.558946 5437 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:33.559911 5437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/data/instance:
uuid: "dc4dcc727a2c4ac7b68a52502d4a839f"
format_stamp: "Formatted at 2025-10-26 15:14:33 on dist-test-slave-8pgz"
server_key: "af6ee9386784c31c89abf6571f254cb3"
server_key_iv: "4d9d76754ce2f90db148cb383ae3c741"
server_key_version: "encryptionkey@0"
I20251026 15:14:33.560231 5437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/wal/instance:
uuid: "dc4dcc727a2c4ac7b68a52502d4a839f"
format_stamp: "Formatted at 2025-10-26 15:14:33 on dist-test-slave-8pgz"
server_key: "af6ee9386784c31c89abf6571f254cb3"
server_key_iv: "4d9d76754ce2f90db148cb383ae3c741"
server_key_version: "encryptionkey@0"
I20251026 15:14:33.561499 5437 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:14:33.562366 5452 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:33.562605 5437 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:14:33.562682 5437 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/wal
uuid: "dc4dcc727a2c4ac7b68a52502d4a839f"
format_stamp: "Formatted at 2025-10-26 15:14:33 on dist-test-slave-8pgz"
server_key: "af6ee9386784c31c89abf6571f254cb3"
server_key_iv: "4d9d76754ce2f90db148cb383ae3c741"
server_key_version: "encryptionkey@0"
I20251026 15:14:33.562739 5437 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:33.577446 5437 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:33.579517 5437 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:33.579629 5437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:33.583443 5437 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.124:33723
I20251026 15:14:33.583498 5504 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.124:33723 every 8 connection(s)
I20251026 15:14:33.583808 5437 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/data/info.pb
I20251026 15:14:33.584426 5505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:33.586238 5505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:14:33.591130 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 5437
I20251026 15:14:33.591245 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761491252308956-15185-0/minicluster-data/master-2/wal/instance
I20251026 15:14:33.591410 15185 external_mini_cluster.cc:1468] Setting key 8544c3124daee936a381dc7d350f6699
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.124@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:14:33.595043 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.587058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.14.212.126:33363)
1026 15:14:33.587171 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.587174 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.587298 (+ 124us) server_negotiation.cc:408] Connection header received
1026 15:14:33.587678 (+ 380us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.587679 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.587692 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.587769 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.588558 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.589035 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.589579 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.589763 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.592159 (+ 2396us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.592163 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.592163 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.592170 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.593661 (+ 1491us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.594111 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.594112 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.594112 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.594143 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.594328 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.594329 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.594329 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.594458 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.594529 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.594831 (+ 302us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.594924 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251026 15:14:33.595202 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.586979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:33363 (local address 127.0.0.1:46148)
1026 15:14:33.587173 (+ 194us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.587211 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:33.587570 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:33.587846 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:33.587850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:33.588176 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:33.588451 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.588454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.589135 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.589135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:33.589491 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.589492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.589611 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.590281 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:33.590300 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:33.592065 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:33.593752 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.593758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.593761 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.594043 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.594216 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.594216 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.594217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.594274 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.594520 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:33.594523 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:33.594671 (+ 148us) client_negotiation.cc:769] Sending connection context
1026 15:14:33.594796 (+ 125us) client_negotiation.cc:240] Negotiation successful
1026 15:14:33.594927 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20251026 15:14:33.596025 5505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } has no permanent_uuid. Determining permanent_uuid...
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.124@KRBTEST.COM
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.124@KRBTEST.COM for kudu/127.14.212.125@KRBTEST.COM
I20251026 15:14:33.599956 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.591805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.14.212.124:33723)
1026 15:14:33.591955 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.591963 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.592107 (+ 144us) server_negotiation.cc:408] Connection header received
1026 15:14:33.592131 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.592134 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.592149 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.592202 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.593115 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.593884 (+ 769us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.594607 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.594825 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.597018 (+ 2193us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.597026 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.597026 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.597033 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.598919 (+ 1886us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.599256 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.599259 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.599259 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.599287 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.599432 (+ 145us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.599433 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.599434 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.599556 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.599701 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.599814 (+ 113us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.599879 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20251026 15:14:33.602263 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.596258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:39319 (local address 127.0.0.1:43186)
1026 15:14:33.596313 (+ 55us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.596322 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:33.596355 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:33.596501 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:33.596502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:33.596649 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:33.596799 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.596801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.597564 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.597565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:33.598038 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.598040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.598115 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.598522 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:33.598526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:33.599714 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:33.601462 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.601463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.601463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.601597 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.601742 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.601742 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.601742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.601776 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.601976 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:33.601978 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:33.602018 (+ 40us) client_negotiation.cc:769] Sending connection context
1026 15:14:33.602081 (+ 63us) client_negotiation.cc:240] Negotiation successful
1026 15:14:33.602174 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20251026 15:14:33.602272 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.596325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.14.212.125:39319)
1026 15:14:33.596399 (+ 74us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.596401 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.596410 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:14:33.596432 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.596433 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.596442 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.596489 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.596879 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.597464 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.598124 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.598302 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.599826 (+ 1524us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.599831 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.599831 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.599838 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.601382 (+ 1544us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.601663 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.601664 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.601664 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.601684 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.601837 (+ 153us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.601838 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.601839 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.601919 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.601975 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.602112 (+ 137us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.602187 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251026 15:14:33.602696 5505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.124@KRBTEST.COM for kudu/127.14.212.124@KRBTEST.COM
I20251026 15:14:33.608319 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.600808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.14.212.126:33363)
1026 15:14:33.600856 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.600858 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.600869 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:14:33.600894 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.600895 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.600906 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.600970 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.601504 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.601925 (+ 421us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.602641 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.602828 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.605565 (+ 2737us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.605570 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.605570 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.605578 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.607176 (+ 1598us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.607502 (+ 326us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.607503 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.607504 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.607528 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.607710 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.607710 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.607711 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.607810 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.607971 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.608113 (+ 142us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.608203 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251026 15:14:33.609596 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.603102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.14.212.124:33723)
1026 15:14:33.603173 (+ 71us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.603176 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.603186 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:14:33.603206 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.603207 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.603220 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.603293 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.603598 (+ 305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.604298 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.605070 (+ 772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.605287 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.606735 (+ 1448us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.606739 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.606739 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.606747 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.608499 (+ 1752us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.608864 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.608865 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.608866 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.608893 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.609060 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.609061 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.609061 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.609142 (+ 81us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.609204 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.609382 (+ 178us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.609510 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":50}
I20251026 15:14:33.609601 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.602948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50218)
1026 15:14:33.602988 (+ 40us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.602997 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:33.603032 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:33.603302 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:33.603302 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:33.603408 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:33.603517 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.603519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.604404 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.604405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:33.604979 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.604982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.605058 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.605482 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:33.605486 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:33.606641 (+ 1155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:33.608589 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.608590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.608590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.608763 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.608964 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.608964 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.608965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.608995 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.609227 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:33.609227 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:33.609263 (+ 36us) client_negotiation.cc:769] Sending connection context
1026 15:14:33.609332 (+ 69us) client_negotiation.cc:240] Negotiation successful
1026 15:14:33.609447 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20251026 15:14:33.611646 5505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f: Bootstrap starting.
I20251026 15:14:33.612419 5505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:33.612676 5505 log.cc:826] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f: Log is configured to *not* fsync() on all Append() calls
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.125@KRBTEST.COM
I20251026 15:14:33.613410 5505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f: No bootstrap required, opened a new log
I20251026 15:14:33.615026 5505 raft_consensus.cc:359] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } }
I20251026 15:14:33.615196 5505 raft_consensus.cc:740] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4dcc727a2c4ac7b68a52502d4a839f, State: Initialized, Role: FOLLOWER
I20251026 15:14:33.615267 5505 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } }
I20251026 15:14:33.615576 5511 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } } }
I20251026 15:14:33.615662 5511 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:14:33.615607 5505 sys_catalog.cc:565] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:33.617237 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.609196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.14.212.125:39319)
1026 15:14:33.609236 (+ 40us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.609239 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.609250 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:14:33.609297 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.609298 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.609306 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.609367 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.610159 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.610768 (+ 609us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.611474 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.611606 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.613834 (+ 2228us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.613840 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.613840 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.613849 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.615765 (+ 1916us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.616122 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.616123 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.616124 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.616156 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.616566 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.616568 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.616568 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.616709 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.616912 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.617013 (+ 101us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.617112 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20251026 15:14:33.618243 5522 catalog_manager.cc:1568] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:14:33.618340 5522 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:14:33.619585 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:33.618824 5437 master_runner.cc:186] Error getting master registration for 127.14.212.126:33363: OK, instance_id { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" instance_seqno: 1761491673215285 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251026 15:14:33.620581 5437 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251026 15:14:33.620625 5325 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46152
I20251026 15:14:33.639479 5428 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } attempt: 3
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.125@KRBTEST.COM for kudu/127.14.212.124@KRBTEST.COM
I20251026 15:14:33.647084 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.639873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50226)
1026 15:14:33.639942 (+ 69us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.639957 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:33.640030 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:33.640220 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:33.640221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:33.640431 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:33.640638 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.640639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.641617 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.641618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:33.642199 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.642201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.642277 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.642895 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:33.642900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:33.644343 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:33.646102 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.646103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.646104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.646258 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.646435 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.646436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.646437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.646469 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.646724 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:33.646725 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:33.646774 (+ 49us) client_negotiation.cc:769] Sending connection context
1026 15:14:33.646839 (+ 65us) client_negotiation.cc:240] Negotiation successful
1026 15:14:33.646966 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20251026 15:14:33.647084 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.639965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.14.212.124:33723)
1026 15:14:33.640058 (+ 93us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.640060 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.640072 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:14:33.640101 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.640102 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.640113 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.640184 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.640732 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.641522 (+ 790us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.642305 (+ 783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.642513 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.644450 (+ 1937us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.644454 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.644455 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.644464 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.645996 (+ 1532us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.646329 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.646330 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.646331 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.646355 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.646528 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.646529 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.646530 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.646656 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.646723 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.646915 (+ 192us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.646999 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251026 15:14:33.649129 5428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069: Bootstrap starting.
I20251026 15:14:33.649952 5428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:33.650224 5428 log.cc:826] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:33.650983 5428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069: No bootstrap required, opened a new log
I20251026 15:14:33.652134 5428 raft_consensus.cc:359] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } }
I20251026 15:14:33.652261 5428 raft_consensus.cc:740] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e18471498f4a4b4ca0ec50589ead2069, State: Initialized, Role: FOLLOWER
I20251026 15:14:33.652323 5428 consensus_queue.cc:260] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } }
I20251026 15:14:33.652611 5428 sys_catalog.cc:565] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:33.652581 5525 sys_catalog.cc:455] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } } }
I20251026 15:14:33.652674 5525 sys_catalog.cc:458] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069 [sys.catalog]: This master's current role is: FOLLOWER
W20251026 15:14:33.654466 5536 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:14:33.654523 5536 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:14:33.654829 5360 master_runner.cc:186] Error getting master registration for 127.14.212.126:33363: OK, instance_id { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" instance_seqno: 1761491673215285 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251026 15:14:33.656553 5360 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251026 15:14:33.657224 5351 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } attempt: 3
Oct 26 15:14:33 dist-test-slave-8pgz krb5kdc[5267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491673, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for kudu/127.14.212.124@KRBTEST.COM
I20251026 15:14:33.665746 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.657584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:33723 (local address 127.0.0.1:50232)
1026 15:14:33.657642 (+ 58us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:33.657654 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:33.657696 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:33.658055 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:33.658057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:33.658272 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:33.658510 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.658513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.659483 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.659484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:33.660063 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:33.660065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.660135 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.660776 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:33.660781 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:33.662473 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:33.664600 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.664601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.664601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.664812 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.665019 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:33.665020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:33.665021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:33.665064 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:33.665364 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:33.665365 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:33.665437 (+ 72us) client_negotiation.cc:769] Sending connection context
1026 15:14:33.665502 (+ 65us) client_negotiation.cc:240] Negotiation successful
1026 15:14:33.665628 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20251026 15:14:33.666306 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.657648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.14.212.124:33723)
1026 15:14:33.657696 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.657699 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.657749 (+ 50us) server_negotiation.cc:408] Connection header received
1026 15:14:33.657850 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.657852 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.657869 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.657957 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.658617 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.659381 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.660333 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.660544 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.662577 (+ 2033us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.662582 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.662582 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.662591 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.664478 (+ 1887us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.664902 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.664903 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.664904 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.664943 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.665134 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.665135 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.665136 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.665268 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.665327 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.666123 (+ 796us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.666212 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251026 15:14:33.668526 5351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2: Bootstrap starting.
I20251026 15:14:33.669245 5351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:33.669507 5351 log.cc:826] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:33.670475 5351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2: No bootstrap required, opened a new log
I20251026 15:14:33.671561 5351 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } }
I20251026 15:14:33.671681 5351 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2, State: Initialized, Role: FOLLOWER
I20251026 15:14:33.671749 5351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } }
I20251026 15:14:33.672058 5351 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:33.672072 5539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } } }
I20251026 15:14:33.672228 5539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:14:33.673636 5325 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 15:14:33.673712 5325 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [term 0 FOLLOWER]: Advancing to term 1
W20251026 15:14:33.673954 5550 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:14:33.674010 5550 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:14:33.674583 5325 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } }
I20251026 15:14:33.674830 5325 leader_election.cc:290] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [CANDIDATE]: Term 1 election: Requested vote from peers e18471498f4a4b4ca0ec50589ead2069 (127.14.212.125:39319), dc4dcc727a2c4ac7b68a52502d4a839f (127.14.212.124:33723)
I20251026 15:14:33.675411 5402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e18471498f4a4b4ca0ec50589ead2069"
I20251026 15:14:33.675535 5402 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:33.676440 5402 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 in term 1.
I20251026 15:14:33.676674 5299 leader_election.cc:304] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2, e18471498f4a4b4ca0ec50589ead2069; no voters:
I20251026 15:14:33.676805 5539 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:33.676918 5539 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [term 1 LEADER]: Becoming Leader. State: Replica: 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2, State: Running, Role: LEADER
I20251026 15:14:33.677006 5539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } }
I20251026 15:14:33.677368 5539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2. Latest consensus state: current_term: 1 leader_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } } }
I20251026 15:14:33.677429 5539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:33.677934 5479 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f"
I20251026 15:14:33.678043 5479 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:33.679307 5479 raft_consensus.cc:2468] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 in term 1.
I20251026 15:14:33.680524 5558 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:33.680814 5558 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:33.682420 5479 raft_consensus.cc:1275] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f [term 1 FOLLOWER]: Refusing update from remote peer 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:14:33.682497 5402 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069 [term 1 FOLLOWER]: Refusing update from remote peer 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:14:33.682688 5539 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:14:33.682912 5539 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:14:33.683879 5511 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2. Latest consensus state: current_term: 1 leader_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } } }
I20251026 15:14:33.683971 5511 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:14:33.683969 5525 sys_catalog.cc:455] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2. Latest consensus state: current_term: 1 leader_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } } }
I20251026 15:14:33.684046 5525 sys_catalog.cc:458] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:14:33.685061 5556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } } }
I20251026 15:14:33.685154 5556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:33.685496 5525 sys_catalog.cc:455] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } } }
I20251026 15:14:33.685563 5525 sys_catalog.cc:458] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:14:33.685858 5559 mvcc.cc:204] Tried to move back new op lower bound from 7215069895400968192 to 7215069895381647360. Current Snapshot: MvccSnapshot[applied={T|T < 7215069895400968192}]
I20251026 15:14:33.686375 5561 mvcc.cc:204] Tried to move back new op lower bound from 7215069895400968192 to 7215069895381647360. Current Snapshot: MvccSnapshot[applied={T|T < 7215069895400968192}]
I20251026 15:14:33.687028 5558 catalog_manager.cc:1357] Generated new cluster ID: 65418738a6204ef5ac9028908b014042
I20251026 15:14:33.687074 5558 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:33.691846 5539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } } }
I20251026 15:14:33.691954 5539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2 [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:33.692382 5511 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9c62ad3e6d4c7e90c67ba5f5d1d8d2" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 33363 } } peers { permanent_uuid: "e18471498f4a4b4ca0ec50589ead2069" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 39319 } } peers { permanent_uuid: "dc4dcc727a2c4ac7b68a52502d4a839f" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 33723 } } }
I20251026 15:14:33.692485 5511 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:14:33.694602 5560 mvcc.cc:204] Tried to move back new op lower bound from 7215069895400968192 to 7215069895381647360. Current Snapshot: MvccSnapshot[applied={T|T < 7215069895400968192}]
I20251026 15:14:33.702533 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.686948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.14.212.124:33723)
1026 15:14:33.687020 (+ 72us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.687023 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.691569 (+ 4546us) server_negotiation.cc:408] Connection header received
1026 15:14:33.692136 (+ 567us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.692137 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.692147 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.692177 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.694258 (+ 2081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.694920 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.697115 (+ 2195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.697298 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.699511 (+ 2213us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.699515 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.699516 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.699523 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.701341 (+ 1818us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.701702 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.701703 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.701703 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.701725 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.701892 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.701893 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.701894 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.702036 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.702110 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.702320 (+ 210us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.702430 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20251026 15:14:33.709529 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.686738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.14.212.126:33363)
1026 15:14:33.686787 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.686790 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.690767 (+ 3977us) server_negotiation.cc:408] Connection header received
1026 15:14:33.690879 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.690880 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.690891 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.690919 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.702495 (+ 11576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.703155 (+ 660us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.703881 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.704053 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.705459 (+ 1406us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.705464 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.705464 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.705472 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.707388 (+ 1916us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.708595 (+ 1207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.708596 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.708596 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.708623 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.708775 (+ 152us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.708776 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.708776 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.708892 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.708978 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.709341 (+ 363us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.709425 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20251026 15:14:33.717212 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:33.690664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.14.212.125:39319)
1026 15:14:33.690707 (+ 43us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.690710 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.693209 (+ 2499us) server_negotiation.cc:408] Connection header received
1026 15:14:33.693311 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.693312 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.693323 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.693352 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.695664 (+ 2312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.696353 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:33.697634 (+ 1281us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:33.697826 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:33.703334 (+ 5508us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:33.703338 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:33.703338 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:33.703347 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:33.708205 (+ 4858us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.710268 (+ 2063us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.710269 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.710270 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.710300 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:33.710488 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:33.710489 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:33.710489 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:33.710618 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:33.710703 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:33.716951 (+ 6248us) server_negotiation.cc:300] Negotiation successful
1026 15:14:33.717100 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251026 15:14:33.719966 5558 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:33.720906 5558 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:33.732406 5558 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7f9c62ad3e6d4c7e90c67ba5f5d1d8d2: Generated new TSK 0
I20251026 15:14:33.732749 5558 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251026 15:14:33.758363 5509 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:33.756766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.14.212.124:33723)
1026 15:14:33.756832 (+ 66us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.756835 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.757530 (+ 695us) server_negotiation.cc:408] Connection header received
1026 15:14:33.757702 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.757704 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.757717 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.757755 (+ 38us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.758298 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50250: BlockingRecv error: recv got EOF from 127.0.0.1:50250 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20251026 15:14:33.759475 5354 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:33.757220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.14.212.126:33363)
1026 15:14:33.757267 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.757269 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.758854 (+ 1585us) server_negotiation.cc:408] Connection header received
1026 15:14:33.758982 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.758984 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.758997 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.759033 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.759413 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46168: BlockingRecv error: recv got EOF from 127.0.0.1:46168 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20251026 15:14:33.760680 5432 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:33.760042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43212 (local address 127.14.212.125:39319)
1026 15:14:33.760089 (+ 47us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.760091 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.760124 (+ 33us) server_negotiation.cc:408] Connection header received
1026 15:14:33.760207 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.760208 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.760219 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.760298 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.760623 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43212: BlockingRecv error: recv got EOF from 127.0.0.1:43212 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20251026 15:14:33.763410 5432 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:33.762450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.14.212.125:39319)
1026 15:14:33.762496 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.762498 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.762779 (+ 281us) server_negotiation.cc:408] Connection header received
1026 15:14:33.762916 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.762929 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.762951 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.762988 (+ 37us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.763361 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43216: BlockingRecv error: recv got EOF from 127.0.0.1:43216 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20251026 15:14:33.767019 5432 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:33.766154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43222 (local address 127.14.212.125:39319)
1026 15:14:33.766195 (+ 41us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.766196 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.766381 (+ 185us) server_negotiation.cc:408] Connection header received
1026 15:14:33.766528 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.766529 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.766541 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.766577 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.766963 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43222: BlockingRecv error: recv got EOF from 127.0.0.1:43222 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20251026 15:14:33.771641 5432 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:33.770675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.14.212.125:39319)
1026 15:14:33.770721 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.770723 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.771021 (+ 298us) server_negotiation.cc:408] Connection header received
1026 15:14:33.771142 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.771143 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.771155 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.771191 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.771589 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43238: BlockingRecv error: recv got EOF from 127.0.0.1:43238 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20251026 15:14:33.778582 5432 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:33.777434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.14.212.125:39319)
1026 15:14:33.777480 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.777483 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.777807 (+ 324us) server_negotiation.cc:408] Connection header received
1026 15:14:33.777944 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.777946 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.777959 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.777996 (+ 37us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.778523 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43242: BlockingRecv error: recv got EOF from 127.0.0.1:43242 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20251026 15:14:33.789387 5432 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:33.788356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.14.212.125:39319)
1026 15:14:33.788405 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.788407 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.788700 (+ 293us) server_negotiation.cc:408] Connection header received
1026 15:14:33.788826 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.788827 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.788841 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.788877 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.789322 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43252: BlockingRecv error: recv got EOF from 127.0.0.1:43252 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20251026 15:14:33.808285 5432 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:33.807212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.14.212.125:39319)
1026 15:14:33.807268 (+ 56us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.807271 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.807577 (+ 306us) server_negotiation.cc:408] Connection header received
1026 15:14:33.807706 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.807707 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.807720 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.807757 (+ 37us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.808225 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43268: BlockingRecv error: recv got EOF from 127.0.0.1:43268 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20251026 15:14:33.842732 5432 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:33.841923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.14.212.125:39319)
1026 15:14:33.841980 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.841983 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.842082 (+ 99us) server_negotiation.cc:408] Connection header received
1026 15:14:33.842148 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.842149 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.842161 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.842215 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.842672 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43278: BlockingRecv error: recv got EOF from 127.0.0.1:43278 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20251026 15:14:33.908937 5432 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:33.908063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.14.212.125:39319)
1026 15:14:33.908118 (+ 55us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:33.908121 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:33.908271 (+ 150us) server_negotiation.cc:408] Connection header received
1026 15:14:33.908319 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:33.908321 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:33.908337 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:33.908430 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:33.908872 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43284: BlockingRecv error: recv got EOF from 127.0.0.1:43284 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20251026 15:14:34.038834 5432 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:34.038213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.14.212.125:39319)
1026 15:14:34.038296 (+ 83us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:34.038298 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:34.038308 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:14:34.038340 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:34.038341 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:34.038351 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:34.038429 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:34.038767 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43296: BlockingRecv error: recv got EOF from 127.0.0.1:43296 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20251026 15:14:34.296914 5432 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:34.296277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43306 (local address 127.14.212.125:39319)
1026 15:14:34.296345 (+ 68us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:34.296347 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:34.296356 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:14:34.296383 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:34.296384 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:34.296394 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:34.296464 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:34.296851 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43306: BlockingRecv error: recv got EOF from 127.0.0.1:43306 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20251026 15:14:34.618816 5522 catalog_manager.cc:1269] Loaded cluster ID: 65418738a6204ef5ac9028908b014042
I20251026 15:14:34.618899 5522 catalog_manager.cc:1562] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f: loading cluster ID for follower catalog manager: success
I20251026 15:14:34.619748 5522 catalog_manager.cc:1584] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f: acquiring CA information for follower catalog manager: success
I20251026 15:14:34.620028 5522 catalog_manager.cc:1612] T 00000000000000000000000000000000 P dc4dcc727a2c4ac7b68a52502d4a839f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 15:14:34.654920 5536 catalog_manager.cc:1269] Loaded cluster ID: 65418738a6204ef5ac9028908b014042
I20251026 15:14:34.655014 5536 catalog_manager.cc:1562] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069: loading cluster ID for follower catalog manager: success
I20251026 15:14:34.655841 5536 catalog_manager.cc:1584] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069: acquiring CA information for follower catalog manager: success
I20251026 15:14:34.656104 5536 catalog_manager.cc:1612] T 00000000000000000000000000000000 P e18471498f4a4b4ca0ec50589ead2069: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 15:14:34.812704 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:34.810361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.14.212.125:39319)
1026 15:14:34.810502 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:34.810504 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:34.810513 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:14:34.810540 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:34.810541 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:34.810581 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:34.810671 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:14:34.811046 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:34.811546 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:34.812166 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:34.812302 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:34.812348 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:14:34.812452 (+ 104us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:14:34.812527 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:34.812606 (+ 79us) server_negotiation.cc:300] Negotiation successful
1026 15:14:34.812629 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20251026 15:14:34.816709 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:34.814518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.14.212.124:33723)
1026 15:14:34.814646 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:34.814649 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:34.814658 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:14:34.814753 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:34.814753 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:34.814779 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:34.814838 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:14:34.815090 (+ 252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:34.815575 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:34.816197 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:34.816329 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:34.816369 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:14:34.816467 (+ 98us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:14:34.816539 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:34.816607 (+ 68us) server_negotiation.cc:300] Negotiation successful
1026 15:14:34.816634 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":53,"threads_started":1}
I20251026 15:14:34.821090 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:34.818683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.14.212.124:33723)
1026 15:14:34.818760 (+ 77us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:34.818762 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:34.818784 (+ 22us) server_negotiation.cc:408] Connection header received
1026 15:14:34.818880 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:34.818881 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:34.818896 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:34.818946 (+ 50us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:14:34.819321 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:34.819813 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:34.820560 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:34.820759 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:34.820782 (+ 23us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:14:34.820812 (+ 30us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:14:34.820867 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:34.820977 (+ 110us) server_negotiation.cc:300] Negotiation successful
1026 15:14:34.821011 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251026 15:14:34.821156 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:34.818533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.14.212.125:39319)
1026 15:14:34.818660 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:34.818662 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:34.818813 (+ 151us) server_negotiation.cc:408] Connection header received
1026 15:14:34.818853 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:34.818854 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:34.818870 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:34.818914 (+ 44us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:14:34.819216 (+ 302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:34.819939 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:34.820667 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:34.820818 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:34.820836 (+ 18us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:14:34.820867 (+ 31us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:14:34.820935 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:34.821047 (+ 112us) server_negotiation.cc:300] Negotiation successful
1026 15:14:34.821084 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20251026 15:14:34.821516 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:34.818537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.14.212.126:33363)
1026 15:14:34.818726 (+ 189us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:34.818728 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:34.818932 (+ 204us) server_negotiation.cc:408] Connection header received
1026 15:14:34.818990 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:34.818992 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:34.819022 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:34.819110 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:14:34.819399 (+ 289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:34.820116 (+ 717us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:34.820758 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:34.820943 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:34.821001 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:14:34.821202 (+ 201us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:14:34.821267 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:34.821411 (+ 144us) server_negotiation.cc:300] Negotiation successful
1026 15:14:34.821441 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":57,"threads_started":1}
I20251026 15:14:34.823396 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 5283
I20251026 15:14:34.828926 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 5360
I20251026 15:14:34.834057 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 5437
2025-10-26T15:14:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3892 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 26 15:14:34 dist-test-slave-8pgz krb5kdc[5650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 26 15:14:34 dist-test-slave-8pgz krb5kdc[5650](info): set up 2 sockets
Oct 26 15:14:34 dist-test-slave-8pgz krb5kdc[5650](info): commencing operation
krb5kdc: starting...
W20251026 15:14:36.840296 15185 mini_kdc.cc:121] Time spent starting KDC: real 1.986s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/test-user.keytab.
Oct 26 15:14:36 dist-test-slave-8pgz krb5kdc[5650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-26T15:14:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-26T15:14:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:36.981930 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:38291
--webserver_interface=127.14.212.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:38291,127.14.212.125:43653,127.14.212.124:41873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251026 15:14:37.061069 5666 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251026 15:14:37.061281 5666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:37.061308 5666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:37.061342 5666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:37.062784 5666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:37.062829 5666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:37.062847 5666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:37.062865 5666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:37.062885 5666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:37.064477 5666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:38291,127.14.212.125:43653,127.14.212.124:41873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.126:38291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.212.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:37.064699 5666 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:37.064888 5666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:37.067498 5674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:37.067545 5671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:37.067541 5672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:37.067881 5666 server_base.cc:1047] running on GCE node
I20251026 15:14:37.068109 5666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:37.068320 5666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:37.069471 5666 hybrid_clock.cc:648] HybridClock initialized: now 1761491677069456 us; error 80 us; skew 500 ppm
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491677, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:37.072225 5666 init.cc:377] Logged in from keytab as kudu/127.14.212.126@KRBTEST.COM (short username kudu)
I20251026 15:14:37.072750 5666 webserver.cc:492] Webserver started at http://127.14.212.126:39627/ using document root <none> and password file <none>
I20251026 15:14:37.072953 5666 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:37.072989 5666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:37.073081 5666 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:37.073994 5666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/data/instance:
uuid: "5fd6820ecf1b47eb8f10ca86affea4ff"
format_stamp: "Formatted at 2025-10-26 15:14:37 on dist-test-slave-8pgz"
server_key: "3b39b47eca48eeb9f85102f13b3137ae"
server_key_iv: "dfa5a91690143514ba65c5295fccf9f5"
server_key_version: "encryptionkey@0"
I20251026 15:14:37.074251 5666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/wal/instance:
uuid: "5fd6820ecf1b47eb8f10ca86affea4ff"
format_stamp: "Formatted at 2025-10-26 15:14:37 on dist-test-slave-8pgz"
server_key: "3b39b47eca48eeb9f85102f13b3137ae"
server_key_iv: "dfa5a91690143514ba65c5295fccf9f5"
server_key_version: "encryptionkey@0"
I20251026 15:14:37.075562 5666 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20251026 15:14:37.076289 5681 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:37.076480 5666 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 15:14:37.076555 5666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/wal
uuid: "5fd6820ecf1b47eb8f10ca86affea4ff"
format_stamp: "Formatted at 2025-10-26 15:14:37 on dist-test-slave-8pgz"
server_key: "3b39b47eca48eeb9f85102f13b3137ae"
server_key_iv: "dfa5a91690143514ba65c5295fccf9f5"
server_key_version: "encryptionkey@0"
I20251026 15:14:37.076606 5666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:37.084065 5666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:37.089126 5666 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:37.089236 5666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:37.093083 5666 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.126:38291
I20251026 15:14:37.093125 5733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.126:38291 every 8 connection(s)
I20251026 15:14:37.093436 5666 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/data/info.pb
I20251026 15:14:37.094049 5734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:37.095865 5734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:14:37.096802 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 5666
I20251026 15:14:37.096894 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-0/wal/instance
I20251026 15:14:37.097054 15185 external_mini_cluster.cc:1468] Setting key 11139e54e062c493d27b28db111b1d84
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491677, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.212.125@KRBTEST.COM; defaulting to no policy
I20251026 15:14:37.105289 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.096551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.14.212.126:38291)
1026 15:14:37.096750 (+ 199us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.096760 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.096788 (+ 28us) server_negotiation.cc:408] Connection header received
1026 15:14:37.097188 (+ 400us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.097192 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.097228 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.097321 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.098034 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.098880 (+ 846us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.099487 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.099734 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.101797 (+ 2063us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.101806 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.101806 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.101814 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.103831 (+ 2017us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.104178 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.104181 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.104182 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.104236 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.104506 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.104507 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.104507 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.104690 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.104790 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.104987 (+ 197us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.105154 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":166,"thread_start_us":114,"threads_started":1}
I20251026 15:14:37.105293 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.096463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:38291 (local address 127.0.0.1:44886)
1026 15:14:37.096609 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.096644 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:37.097042 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:37.097331 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:37.097333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:37.097636 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:37.097917 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.097921 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.099014 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.099016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:37.099392 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.099393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.099498 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.100053 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:37.100070 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:37.101699 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:37.103929 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.103936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.103938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.104099 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.104337 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.104338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.104339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.104417 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.104779 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:37.104784 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:37.104845 (+ 61us) client_negotiation.cc:769] Sending connection context
1026 15:14:37.104928 (+ 83us) client_negotiation.cc:240] Negotiation successful
1026 15:14:37.105076 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20251026 15:14:37.106118 5734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:14:37.106493 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.106356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:43653 (local address 127.0.0.1:41918)
1026 15:14:37.106429 (+ 73us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.106472 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20251026 15:14:37.106609 5683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.125:43653: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.125@KRBTEST.COM" created.
I20251026 15:14:37.107067 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.106962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:43653 (local address 127.0.0.1:41928)
1026 15:14:37.107013 (+ 51us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.107036 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20251026 15:14:37.107239 5734 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.125:43653: Network error: Client connection negotiation failed: client connection to 127.14.212.125:43653: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:37.145678 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.125:43653
--webserver_interface=127.14.212.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:38291,127.14.212.125:43653,127.14.212.124:41873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:14:37.156484 5734 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } attempt: 1
I20251026 15:14:37.156975 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.156844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:43653 (local address 127.0.0.1:41934)
1026 15:14:37.156904 (+ 60us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.156938 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20251026 15:14:37.157426 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.157284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:43653 (local address 127.0.0.1:41946)
1026 15:14:37.157347 (+ 63us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.157391 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20251026 15:14:37.157601 5734 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.125:43653: Network error: Client connection negotiation failed: client connection to 127.14.212.125:43653: connect: Connection refused (error 111)
I20251026 15:14:37.225812 5734 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } attempt: 2
I20251026 15:14:37.226311 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.226167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:43653 (local address 127.0.0.1:41954)
1026 15:14:37.226232 (+ 65us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.226277 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20251026 15:14:37.226711 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.226591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:43653 (local address 127.0.0.1:41962)
1026 15:14:37.226649 (+ 58us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.226688 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.125:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20251026 15:14:37.226557 5743 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251026 15:14:37.226775 5743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:37.226804 5743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:37.226852 5743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:37.226837 5734 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.125:43653: Network error: Client connection negotiation failed: client connection to 127.14.212.125:43653: connect: Connection refused (error 111)
W20251026 15:14:37.228349 5743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:37.228394 5743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:37.228408 5743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:37.228421 5743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:37.228433 5743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:37.230042 5743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:38291,127.14.212.125:43653,127.14.212.124:41873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.125:43653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.212.125
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:37.230252 5743 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:37.230461 5743 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 15:14:37.232977 5748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:37.233033 5743 server_base.cc:1047] running on GCE node
W20251026 15:14:37.232977 5749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:37.232977 5751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:37.233417 5743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:37.233647 5743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:37.234828 5743 hybrid_clock.cc:648] HybridClock initialized: now 1761491677234803 us; error 38 us; skew 500 ppm
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491677, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:37.238247 5743 init.cc:377] Logged in from keytab as kudu/127.14.212.125@KRBTEST.COM (short username kudu)
I20251026 15:14:37.238987 5743 webserver.cc:492] Webserver started at http://127.14.212.125:34549/ using document root <none> and password file <none>
I20251026 15:14:37.239202 5743 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:37.239233 5743 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:37.239344 5743 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:37.240368 5743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/data/instance:
uuid: "6666bc18d4794cf698153614633835c3"
format_stamp: "Formatted at 2025-10-26 15:14:37 on dist-test-slave-8pgz"
server_key: "cc8ab18df619016998b806934d1155d9"
server_key_iv: "8d2b14bc0d6eefec3d997b4db4556245"
server_key_version: "encryptionkey@0"
I20251026 15:14:37.240638 5743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/wal/instance:
uuid: "6666bc18d4794cf698153614633835c3"
format_stamp: "Formatted at 2025-10-26 15:14:37 on dist-test-slave-8pgz"
server_key: "cc8ab18df619016998b806934d1155d9"
server_key_iv: "8d2b14bc0d6eefec3d997b4db4556245"
server_key_version: "encryptionkey@0"
I20251026 15:14:37.241858 5743 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20251026 15:14:37.242640 5758 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:37.242897 5743 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:14:37.242993 5743 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/wal
uuid: "6666bc18d4794cf698153614633835c3"
format_stamp: "Formatted at 2025-10-26 15:14:37 on dist-test-slave-8pgz"
server_key: "cc8ab18df619016998b806934d1155d9"
server_key_iv: "8d2b14bc0d6eefec3d997b4db4556245"
server_key_version: "encryptionkey@0"
I20251026 15:14:37.243067 5743 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:37.252110 5743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:37.254232 5743 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:37.254354 5743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:37.258160 5743 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.125:43653
I20251026 15:14:37.258232 5810 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.125:43653 every 8 connection(s)
I20251026 15:14:37.258533 5743 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/data/info.pb
I20251026 15:14:37.259166 5811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:37.260183 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 5743
I20251026 15:14:37.260282 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-1/wal/instance
I20251026 15:14:37.260417 15185 external_mini_cluster.cc:1468] Setting key e6a09ba7dc332b43b2922cb9673b7ff3
I20251026 15:14:37.261127 5811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } has no permanent_uuid. Determining permanent_uuid...
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491677, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.125@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.212.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.212.124@KRBTEST.COM" created.
I20251026 15:14:37.270444 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.261909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44902 (local address 127.14.212.126:38291)
1026 15:14:37.261963 (+ 54us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.261966 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.262165 (+ 199us) server_negotiation.cc:408] Connection header received
1026 15:14:37.262608 (+ 443us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.262610 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.262627 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.262698 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.263514 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.263996 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.264551 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.264756 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.267165 (+ 2409us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.267171 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.267172 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.267179 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.269135 (+ 1956us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.269540 (+ 405us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.269541 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.269542 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.269571 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.269786 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.269787 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.269787 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.269882 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.269955 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.270231 (+ 276us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.270326 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20251026 15:14:37.270583 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.261824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:38291 (local address 127.0.0.1:44902)
1026 15:14:37.262013 (+ 189us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.262048 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:37.262510 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:37.262772 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:37.262776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:37.263105 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:37.263393 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.263397 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.264102 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.264103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:37.264464 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.264466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.264593 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.265228 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:37.265247 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:37.267051 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:37.269238 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.269245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.269248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.269464 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.269644 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.269645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.269645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.269721 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.269954 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:37.269961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:37.270100 (+ 139us) client_negotiation.cc:769] Sending connection context
1026 15:14:37.270198 (+ 98us) client_negotiation.cc:240] Negotiation successful
1026 15:14:37.270339 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20251026 15:14:37.271435 5811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } has no permanent_uuid. Determining permanent_uuid...
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491677, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.125@KRBTEST.COM for kudu/127.14.212.125@KRBTEST.COM
I20251026 15:14:37.279124 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.271725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:43653 (local address 127.0.0.1:41978)
1026 15:14:37.271776 (+ 51us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.271787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:37.271828 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:37.272086 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:37.272087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:37.272216 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:37.272343 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.272345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.273200 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.273201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:37.273532 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.273534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.273599 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.274219 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:37.274223 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:37.275772 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:37.277987 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.277988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.277989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.278186 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.278487 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.278488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.278488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.278520 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.278811 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:37.278812 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:37.278857 (+ 45us) client_negotiation.cc:769] Sending connection context
1026 15:14:37.278946 (+ 89us) client_negotiation.cc:240] Negotiation successful
1026 15:14:37.279051 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20251026 15:14:37.279224 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.271791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.14.212.125:43653)
1026 15:14:37.271957 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.271964 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.271978 (+ 14us) server_negotiation.cc:408] Connection header received
1026 15:14:37.271997 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.272000 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.272012 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.272068 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.272420 (+ 352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.273126 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.273607 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.273828 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.275892 (+ 2064us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.275900 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.275900 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.275907 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.277795 (+ 1888us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.278273 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.278275 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.278275 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.278309 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.278592 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.278592 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.278593 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.278706 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.278787 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.278977 (+ 190us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.279110 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":127,"thread_start_us":105,"threads_started":1}
I20251026 15:14:37.279888 5811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:14:37.280282 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.280151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41873 (local address 127.0.0.1:45254)
1026 15:14:37.280208 (+ 57us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.280261 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20251026 15:14:37.280413 5759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
I20251026 15:14:37.281141 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.281008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41873 (local address 127.0.0.1:45270)
1026 15:14:37.281082 (+ 74us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.281112 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20251026 15:14:37.281337 5811 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:41873: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.212.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.212.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.212.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.212.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.212.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.212.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab.
I20251026 15:14:37.309224 15185 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXR9jZt/build/release/bin/kudu
/tmp/dist-test-taskXR9jZt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.124:41873
--webserver_interface=127.14.212.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.212.84:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.212.126:38291,127.14.212.125:43653,127.14.212.124:41873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251026 15:14:37.318028 5734 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } attempt: 3
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491677, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for kudu/127.14.212.125@KRBTEST.COM
I20251026 15:14:37.325513 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.318434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:43653 (local address 127.0.0.1:41982)
1026 15:14:37.318496 (+ 62us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.318512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:37.318584 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:37.318748 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:37.318749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:37.318959 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:37.319172 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.319174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.319835 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.319837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:37.320419 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.320421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.320495 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.321046 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:37.321051 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:37.322536 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:37.324518 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.324519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.324520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.324666 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.324849 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.324849 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.324850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.324873 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.325180 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:37.325181 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:37.325254 (+ 73us) client_negotiation.cc:769] Sending connection context
1026 15:14:37.325329 (+ 75us) client_negotiation.cc:240] Negotiation successful
1026 15:14:37.325428 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20251026 15:14:37.325549 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.318530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.14.212.125:43653)
1026 15:14:37.318587 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.318589 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.318599 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:14:37.318647 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.318648 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.318658 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.318737 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.319257 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.319735 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.320499 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.320617 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.322637 (+ 2020us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.322642 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.322642 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.322651 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.324421 (+ 1770us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.324752 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.324754 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.324754 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.324786 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.324927 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.324935 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.324936 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.325071 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.325153 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.325368 (+ 215us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.325461 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20251026 15:14:37.326083 5734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:14:37.326474 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.326333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41873 (local address 127.0.0.1:45286)
1026 15:14:37.326383 (+ 50us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.326445 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20251026 15:14:37.326804 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.326722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41873 (local address 127.0.0.1:45296)
1026 15:14:37.326765 (+ 43us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.326788 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20251026 15:14:37.326942 5734 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:41873: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
I20251026 15:14:37.330510 5811 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } attempt: 1
I20251026 15:14:37.330950 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.330802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41873 (local address 127.0.0.1:45308)
1026 15:14:37.330869 (+ 67us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.330922 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20251026 15:14:37.331306 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.331215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41873 (local address 127.0.0.1:45324)
1026 15:14:37.331259 (+ 44us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.331283 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20251026 15:14:37.331405 5811 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:41873: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
I20251026 15:14:37.358153 5734 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } attempt: 1
I20251026 15:14:37.358722 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.358576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41873 (local address 127.0.0.1:45334)
1026 15:14:37.358643 (+ 67us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.358692 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20251026 15:14:37.359135 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.359028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41873 (local address 127.0.0.1:45338)
1026 15:14:37.359083 (+ 55us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.359110 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20251026 15:14:37.359246 5734 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:41873: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
W20251026 15:14:37.393568 5820 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251026 15:14:37.393784 5820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251026 15:14:37.393810 5820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251026 15:14:37.393844 5820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251026 15:14:37.395325 5820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251026 15:14:37.395380 5820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251026 15:14:37.395399 5820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251026 15:14:37.395426 5820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251026 15:14:37.395447 5820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251026 15:14:37.397130 5820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.212.84:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.212.126:38291,127.14.212.125:43653,127.14.212.124:41873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.212.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.212.124:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.212.124
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type RELEASE
built by None at 26 Oct 2025 14:43:16 UTC on 5fd53c4cbb9d
build id 8727
I20251026 15:14:37.397396 5820 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 15:14:37.397616 5820 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 15:14:37.399735 5811 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } attempt: 2
W20251026 15:14:37.400245 5826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 15:14:37.400352 5828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:37.400323 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.400061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41873 (local address 127.0.0.1:45342)
1026 15:14:37.400106 (+ 45us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.400279 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20251026 15:14:37.400449 5825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 15:14:37.400614 5820 server_base.cc:1047] running on GCE node
I20251026 15:14:37.400836 5820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251026 15:14:37.400871 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.400728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41873 (local address 127.0.0.1:45350)
1026 15:14:37.400809 (+ 81us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.400840 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20251026 15:14:37.400961 5811 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.14.212.124:41873: Network error: Client connection negotiation failed: client connection to 127.14.212.124:41873: connect: Connection refused (error 111)
I20251026 15:14:37.401118 5820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251026 15:14:37.402283 5820 hybrid_clock.cc:648] HybridClock initialized: now 1761491677402240 us; error 64 us; skew 500 ppm
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491677, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251026 15:14:37.405082 5820 init.cc:377] Logged in from keytab as kudu/127.14.212.124@KRBTEST.COM (short username kudu)
I20251026 15:14:37.405675 5820 webserver.cc:492] Webserver started at http://127.14.212.124:36619/ using document root <none> and password file <none>
I20251026 15:14:37.405895 5820 fs_manager.cc:362] Metadata directory not provided
I20251026 15:14:37.405937 5820 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 15:14:37.406039 5820 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 15:14:37.407100 5820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/data/instance:
uuid: "b93b0aaa97de470490732cd45976a905"
format_stamp: "Formatted at 2025-10-26 15:14:37 on dist-test-slave-8pgz"
server_key: "d4399496d0caff6283a93bcde1bd051b"
server_key_iv: "93217e5ce60a318541b45c167177e5f5"
server_key_version: "encryptionkey@0"
I20251026 15:14:37.407439 5820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/wal/instance:
uuid: "b93b0aaa97de470490732cd45976a905"
format_stamp: "Formatted at 2025-10-26 15:14:37 on dist-test-slave-8pgz"
server_key: "d4399496d0caff6283a93bcde1bd051b"
server_key_iv: "93217e5ce60a318541b45c167177e5f5"
server_key_version: "encryptionkey@0"
I20251026 15:14:37.408776 5820 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 15:14:37.409566 5835 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 15:14:37.409785 5820 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 15:14:37.409861 5820 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/data,/tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/wal
uuid: "b93b0aaa97de470490732cd45976a905"
format_stamp: "Formatted at 2025-10-26 15:14:37 on dist-test-slave-8pgz"
server_key: "d4399496d0caff6283a93bcde1bd051b"
server_key_iv: "93217e5ce60a318541b45c167177e5f5"
server_key_version: "encryptionkey@0"
I20251026 15:14:37.409915 5820 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 15:14:37.417013 5820 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 15:14:37.419072 5820 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 15:14:37.419176 5820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 15:14:37.422971 5820 rpc_server.cc:307] RPC server started. Bound to: 127.14.212.124:41873
I20251026 15:14:37.423033 5887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.212.124:41873 every 8 connection(s)
I20251026 15:14:37.423307 5820 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/data/info.pb
I20251026 15:14:37.423944 5888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 15:14:37.424036 15185 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXR9jZt/build/release/bin/kudu as pid 5820
I20251026 15:14:37.424141 15185 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXR9jZt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761491252308956-15185-0/minicluster-data/master-2/wal/instance
I20251026 15:14:37.424302 15185 external_mini_cluster.cc:1468] Setting key fe13bebcfae0d548a98311e7cb972f31
I20251026 15:14:37.426903 5888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } has no permanent_uuid. Determining permanent_uuid...
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.124@KRBTEST.COM
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491677, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.124@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:14:37.434412 5734 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } attempt: 2
I20251026 15:14:37.434787 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.427507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.14.212.126:38291)
1026 15:14:37.427559 (+ 52us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.427561 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.427678 (+ 117us) server_negotiation.cc:408] Connection header received
1026 15:14:37.427766 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.427767 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.427777 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.427824 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.428488 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.429025 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.429662 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.429860 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.432121 (+ 2261us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.432125 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.432125 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.432131 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.433658 (+ 1527us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.434011 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.434012 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.434012 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.434035 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.434210 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.434210 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.434211 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.434303 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.434369 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.434586 (+ 217us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.434665 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20251026 15:14:37.435143 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.424910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.14.212.124:41873)
1026 15:14:37.425068 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.425082 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.425109 (+ 27us) server_negotiation.cc:408] Connection header received
1026 15:14:37.426171 (+ 1062us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.426180 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.426245 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.426442 (+ 197us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.427271 (+ 829us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.428169 (+ 898us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.428869 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.429122 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.431593 (+ 2471us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.431603 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.431604 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.431613 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.433514 (+ 1901us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.433987 (+ 473us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.433990 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.433991 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.434039 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.434207 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.434208 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.434208 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.434428 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.434576 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.434817 (+ 241us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.434960 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20251026 15:14:37.435248 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.427433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.126:38291 (local address 127.0.0.1:44904)
1026 15:14:37.427587 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.427610 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:37.427676 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:37.427854 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:37.427856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:37.428158 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:37.428395 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.428397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.429118 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.429119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:37.429570 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.429572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.429637 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.430108 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:37.430124 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:37.432030 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:37.433763 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.433767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.433770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.433925 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.434103 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.434104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.434104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.434147 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.434364 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:37.434368 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:37.434444 (+ 76us) client_negotiation.cc:769] Sending connection context
1026 15:14:37.435041 (+ 597us) client_negotiation.cc:240] Negotiation successful
1026 15:14:37.435126 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20251026 15:14:37.435673 5888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } has no permanent_uuid. Determining permanent_uuid...
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491677, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.126@KRBTEST.COM for kudu/127.14.212.124@KRBTEST.COM
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491677, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.124@KRBTEST.COM for kudu/127.14.212.125@KRBTEST.COM
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.126@KRBTEST.COM
I20251026 15:14:37.442435 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.435945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.125:43653 (local address 127.0.0.1:41990)
1026 15:14:37.436057 (+ 112us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.436067 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:37.436108 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:37.436244 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:37.436245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:37.436395 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:37.436601 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.436603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.437535 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.437536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:37.437869 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.437871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.437942 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.438386 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:37.438416 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:37.439823 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:37.441595 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.441596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.441597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.441735 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.441883 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.441884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.441884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.441911 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.442116 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:37.442117 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:37.442158 (+ 41us) client_negotiation.cc:769] Sending connection context
1026 15:14:37.442222 (+ 64us) client_negotiation.cc:240] Negotiation successful
1026 15:14:37.442318 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20251026 15:14:37.442466 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.436027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.14.212.125:43653)
1026 15:14:37.436067 (+ 40us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.436070 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.436095 (+ 25us) server_negotiation.cc:408] Connection header received
1026 15:14:37.436175 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.436176 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.436185 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.436241 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.436732 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.437429 (+ 697us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.437952 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.438138 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.439931 (+ 1793us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.439935 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.439935 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.439943 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.441518 (+ 1575us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.441805 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.441806 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.441806 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.441826 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.441972 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.441972 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.441973 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.442059 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.442117 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.442255 (+ 138us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.442344 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20251026 15:14:37.442909 5888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } has no permanent_uuid. Determining permanent_uuid...
I20251026 15:14:37.443910 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.436501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44918 (local address 127.14.212.126:38291)
1026 15:14:37.436543 (+ 42us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.436544 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.436553 (+ 9us) server_negotiation.cc:408] Connection header received
1026 15:14:37.436573 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.436574 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.436584 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.436639 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.437397 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.438063 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.438635 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.438828 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.441098 (+ 2270us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.441102 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.441102 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.441109 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.442583 (+ 1474us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.442994 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.442995 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.442996 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.443019 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.443196 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.443197 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.443198 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.443313 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.443459 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.443653 (+ 194us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.443728 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251026 15:14:37.444063 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.434690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41873 (local address 127.0.0.1:45368)
1026 15:14:37.434752 (+ 62us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.434764 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:37.434803 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:37.435475 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:37.435476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:37.435630 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:37.435819 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.435822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.436602 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.436603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:37.436993 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.436995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.437055 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.437544 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:37.437548 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:37.439823 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:37.443052 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.443053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.443054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.443232 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.443403 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.443403 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.443404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.443430 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.443708 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:37.443709 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:37.443764 (+ 55us) client_negotiation.cc:769] Sending connection context
1026 15:14:37.443813 (+ 49us) client_negotiation.cc:240] Negotiation successful
1026 15:14:37.443905 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20251026 15:14:37.444205 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.434911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.14.212.124:41873)
1026 15:14:37.435085 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.435088 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.435100 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:14:37.435123 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.435124 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.435137 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.435441 (+ 304us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.435908 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.436481 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.437072 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.437275 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.439925 (+ 2650us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.439928 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.439929 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.439936 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.442976 (+ 3040us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.443305 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.443306 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.443307 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.443327 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.443498 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.443498 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.443499 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.443618 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.443960 (+ 342us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.444005 (+ 45us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.444089 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20251026 15:14:37.445977 5734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff: Bootstrap starting.
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491677, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.124@KRBTEST.COM for kudu/127.14.212.124@KRBTEST.COM
I20251026 15:14:37.446938 5734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:37.447269 5734 log.cc:826] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:37.448081 5734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff: No bootstrap required, opened a new log
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.212.125@KRBTEST.COM
I20251026 15:14:37.449793 5734 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } }
I20251026 15:14:37.449928 5734 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd6820ecf1b47eb8f10ca86affea4ff, State: Initialized, Role: FOLLOWER
I20251026 15:14:37.450001 5734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } }
I20251026 15:14:37.450346 5734 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:37.450305 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } } }
I20251026 15:14:37.450558 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:14:37.450773 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.443186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41873 (local address 127.0.0.1:45378)
1026 15:14:37.443224 (+ 38us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.443321 (+ 97us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:37.443360 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:37.443471 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:37.443472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:37.443607 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:37.443771 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.443773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.445098 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.445098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:37.445412 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.445413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.445475 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.445885 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:37.445890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:37.447255 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:37.449247 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.449248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.449249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.449427 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.449593 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.449594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.449594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.449617 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.449908 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:37.449909 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:37.449965 (+ 56us) client_negotiation.cc:769] Sending connection context
1026 15:14:37.450044 (+ 79us) client_negotiation.cc:240] Negotiation successful
1026 15:14:37.450678 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20251026 15:14:37.451134 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.443267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45378 (local address 127.14.212.124:41873)
1026 15:14:37.443310 (+ 43us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.443312 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.443384 (+ 72us) server_negotiation.cc:408] Connection header received
1026 15:14:37.443404 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.443405 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.443414 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.443468 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.443841 (+ 373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.445013 (+ 1172us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.445486 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.445976 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.447363 (+ 1387us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.447366 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.447367 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.447373 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.449153 (+ 1780us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.449502 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.449502 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.449503 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.449525 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.449700 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.449700 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.449701 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.449823 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.449893 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.450577 (+ 684us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.451019 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20251026 15:14:37.452329 5888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905: Bootstrap starting.
W20251026 15:14:37.452677 5906 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:14:37.452723 5906 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:14:37.452957 5888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:37.453207 5888 log.cc:826] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:37.454386 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.444580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.14.212.125:43653)
1026 15:14:37.444613 (+ 33us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.444614 (+ 1us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.444622 (+ 8us) server_negotiation.cc:408] Connection header received
1026 15:14:37.444662 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.444663 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.444672 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.444738 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.445516 (+ 778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.446135 (+ 619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.446890 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.447072 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.449278 (+ 2206us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.449282 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.449282 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.449289 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.451094 (+ 1805us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.453514 (+ 2420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.453516 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.453516 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.453545 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.453703 (+ 158us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.453703 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.453704 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.453813 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.453942 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.454154 (+ 212us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.454263 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20251026 15:14:37.453429 5666 master_runner.cc:186] Error getting master registration for 127.14.212.125:43653: OK, instance_id { permanent_uuid: "6666bc18d4794cf698153614633835c3" instance_seqno: 1761491677257072 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251026 15:14:37.455134 5666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251026 15:14:37.456055 5888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905: No bootstrap required, opened a new log
I20251026 15:14:37.456666 15185 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251026 15:14:37.457594 5708 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44918
I20251026 15:14:37.457543 5888 raft_consensus.cc:359] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } }
I20251026 15:14:37.457682 5888 raft_consensus.cc:740] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b93b0aaa97de470490732cd45976a905, State: Initialized, Role: FOLLOWER
I20251026 15:14:37.457700 5708 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 15:14:37.457744 5708 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:37.457743 5888 consensus_queue.cc:260] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } }
I20251026 15:14:37.458067 5888 sys_catalog.cc:565] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:37.458041 5909 sys_catalog.cc:455] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } } }
I20251026 15:14:37.458212 5909 sys_catalog.cc:458] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:14:37.458726 5708 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } }
I20251026 15:14:37.459015 5708 leader_election.cc:290] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [CANDIDATE]: Term 1 election: Requested vote from peers 6666bc18d4794cf698153614633835c3 (127.14.212.125:43653), b93b0aaa97de470490732cd45976a905 (127.14.212.124:41873)
I20251026 15:14:37.459367 5862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b93b0aaa97de470490732cd45976a905"
I20251026 15:14:37.459466 5862 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:37.460355 5862 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fd6820ecf1b47eb8f10ca86affea4ff in term 1.
I20251026 15:14:37.460621 5682 leader_election.cc:304] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fd6820ecf1b47eb8f10ca86affea4ff, b93b0aaa97de470490732cd45976a905; no voters:
I20251026 15:14:37.460738 5895 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [term 1 FOLLOWER]: Leader election won for term 1
I20251026 15:14:37.460961 5895 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [term 1 LEADER]: Becoming Leader. State: Replica: 5fd6820ecf1b47eb8f10ca86affea4ff, State: Running, Role: LEADER
I20251026 15:14:37.461066 5895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } }
I20251026 15:14:37.461544 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fd6820ecf1b47eb8f10ca86affea4ff. Latest consensus state: current_term: 1 leader_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } } }
I20251026 15:14:37.461627 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:37.461834 5929 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 15:14:37.462024 5929 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 15:14:37.463382 5862 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905 [term 1 FOLLOWER]: Refusing update from remote peer 5fd6820ecf1b47eb8f10ca86affea4ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:14:37.463609 5895 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:14:37.462879 5820 master_runner.cc:186] Error getting master registration for 127.14.212.125:43653: OK, instance_id { permanent_uuid: "6666bc18d4794cf698153614633835c3" instance_seqno: 1761491677257072 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251026 15:14:37.464903 5820 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251026 15:14:37.465683 5909 sys_catalog.cc:455] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fd6820ecf1b47eb8f10ca86affea4ff. Latest consensus state: current_term: 1 leader_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } } }
I20251026 15:14:37.465793 5909 sys_catalog.cc:458] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905 [sys.catalog]: This master's current role is: FOLLOWER
W20251026 15:14:37.468320 5926 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:14:37.468372 5926 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:14:37.474107 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.466817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45382 (local address 127.14.212.124:41873)
1026 15:14:37.466874 (+ 57us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.466876 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.467085 (+ 209us) server_negotiation.cc:408] Connection header received
1026 15:14:37.467185 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.467186 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.467197 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.467227 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.468069 (+ 842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.468756 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.469464 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.469637 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.471066 (+ 1429us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.471070 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.471071 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.471078 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.472895 (+ 1817us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.473287 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.473289 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.473289 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.473312 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.473478 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.473479 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.473479 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.473597 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.473699 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.473917 (+ 218us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.474009 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20251026 15:14:37.476385 5927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } } }
I20251026 15:14:37.476506 5927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:37.476573 5930 mvcc.cc:204] Tried to move back new op lower bound from 7215069910888136704 to 7215069910881525760. Current Snapshot: MvccSnapshot[applied={T|T < 7215069910888136704}]
I20251026 15:14:37.477137 5929 catalog_manager.cc:1357] Generated new cluster ID: e22152a97d3640889552496293cea12c
I20251026 15:14:37.477113 5909 sys_catalog.cc:455] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } } }
I20251026 15:14:37.477202 5909 sys_catalog.cc:458] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:14:37.477416 5931 mvcc.cc:204] Tried to move back new op lower bound from 7215069910888136704 to 7215069910881525760. Current Snapshot: MvccSnapshot[applied={T|T < 7215069910888136704}]
I20251026 15:14:37.478051 5929 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 15:14:37.478533 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.460515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42008 (local address 127.14.212.125:43653)
1026 15:14:37.460559 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.460561 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.461246 (+ 685us) server_negotiation.cc:408] Connection header received
1026 15:14:37.461375 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.461376 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.461399 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.461429 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.463335 (+ 1906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.463982 (+ 647us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.464679 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.464870 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.466301 (+ 1431us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.466305 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.466306 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.466314 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.468276 (+ 1962us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.468715 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.468717 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.468717 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.468748 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.477708 (+ 8960us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.477711 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.477712 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.477885 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.478012 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.478297 (+ 285us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.478433 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251026 15:14:37.485317 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.460998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.14.212.126:38291)
1026 15:14:37.461047 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.461049 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.466905 (+ 5856us) server_negotiation.cc:408] Connection header received
1026 15:14:37.466934 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.466936 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.466957 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.467015 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.467794 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.468434 (+ 640us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.475003 (+ 6569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.475222 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.476986 (+ 1764us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.476989 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.476989 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.476997 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.483806 (+ 6809us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.484226 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.484227 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.484228 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.484255 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.484406 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.484407 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.484407 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.484522 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.484619 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.485123 (+ 504us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.485218 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20251026 15:14:37.492096 5811 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } attempt: 3
Oct 26 15:14:37 dist-test-slave-8pgz krb5kdc[5650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761491677, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.212.125@KRBTEST.COM for kudu/127.14.212.124@KRBTEST.COM
I20251026 15:14:37.500883 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.492401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.212.124:41873 (local address 127.0.0.1:45398)
1026 15:14:37.492455 (+ 54us) negotiation.cc:107] Waiting for socket to connect
1026 15:14:37.492466 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1026 15:14:37.492508 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1026 15:14:37.492676 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1026 15:14:37.492676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
1026 15:14:37.492857 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1026 15:14:37.493060 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.493063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.493964 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.493966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1026 15:14:37.494531 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1026 15:14:37.494533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.494605 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.495177 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1026 15:14:37.495182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1026 15:14:37.496748 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1026 15:14:37.499928 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.499930 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.499930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.500117 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.500275 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1026 15:14:37.500275 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1026 15:14:37.500276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1026 15:14:37.500303 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1026 15:14:37.500553 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1026 15:14:37.500554 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1026 15:14:37.500599 (+ 45us) client_negotiation.cc:769] Sending connection context
1026 15:14:37.500674 (+ 75us) client_negotiation.cc:240] Negotiation successful
1026 15:14:37.500778 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20251026 15:14:37.501190 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:37.492457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.14.212.124:41873)
1026 15:14:37.492501 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.492503 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.492513 (+ 10us) server_negotiation.cc:408] Connection header received
1026 15:14:37.492578 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.492579 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.492591 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.492672 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.493139 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.493850 (+ 711us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:37.494617 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:37.494801 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:37.496850 (+ 2049us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1026 15:14:37.496869 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1026 15:14:37.496870 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1026 15:14:37.496878 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1026 15:14:37.499828 (+ 2950us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.500190 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.500191 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.500192 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.500214 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1026 15:14:37.500360 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1026 15:14:37.500360 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1026 15:14:37.500361 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1026 15:14:37.500477 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1026 15:14:37.500531 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:37.501018 (+ 487us) server_negotiation.cc:300] Negotiation successful
1026 15:14:37.501103 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20251026 15:14:37.502806 5811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3: Bootstrap starting.
I20251026 15:14:37.503727 5811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3: Neither blocks nor log segments found. Creating new log.
I20251026 15:14:37.504022 5811 log.cc:826] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3: Log is configured to *not* fsync() on all Append() calls
I20251026 15:14:37.505272 5811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3: No bootstrap required, opened a new log
I20251026 15:14:37.506927 5811 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } }
I20251026 15:14:37.507051 5811 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6666bc18d4794cf698153614633835c3, State: Initialized, Role: FOLLOWER
I20251026 15:14:37.507109 5811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } }
I20251026 15:14:37.507390 5811 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 15:14:37.509117 5785 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6666bc18d4794cf698153614633835c3"
I20251026 15:14:37.509224 5785 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 15:14:37.510200 5785 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fd6820ecf1b47eb8f10ca86affea4ff in term 1.
I20251026 15:14:37.511052 5784 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3 [term 1 FOLLOWER]: Refusing update from remote peer 5fd6820ecf1b47eb8f10ca86affea4ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 15:14:37.511531 5895 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 15:14:37.513195 5950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } } }
I20251026 15:14:37.513370 5950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:14:37.513465 5950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fd6820ecf1b47eb8f10ca86affea4ff. Latest consensus state: current_term: 1 leader_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } } }
I20251026 15:14:37.513651 5950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 15:14:37.513734 5950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } } }
I20251026 15:14:37.513844 5950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3 [sys.catalog]: This master's current role is: FOLLOWER
W20251026 15:14:37.515234 5949 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 15:14:37.515342 5949 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 15:14:37.515794 5951 mvcc.cc:204] Tried to move back new op lower bound from 7215069910888136704 to 7215069910881525760. Current Snapshot: MvccSnapshot[applied={T|T < 7215069910888136704}]
I20251026 15:14:37.522583 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6820ecf1b47eb8f10ca86affea4ff" member_type: VOTER last_known_addr { host: "127.14.212.126" port: 38291 } } peers { permanent_uuid: "6666bc18d4794cf698153614633835c3" member_type: VOTER last_known_addr { host: "127.14.212.125" port: 43653 } } peers { permanent_uuid: "b93b0aaa97de470490732cd45976a905" member_type: VOTER last_known_addr { host: "127.14.212.124" port: 41873 } } }
I20251026 15:14:37.522708 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff [sys.catalog]: This master's current role is: LEADER
I20251026 15:14:37.524104 5929 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 15:14:37.524901 5929 catalog_manager.cc:1514] Loading token signing keys...
I20251026 15:14:37.534873 5929 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 5fd6820ecf1b47eb8f10ca86affea4ff: Generated new TSK 0
I20251026 15:14:37.535104 5929 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251026 15:14:37.562286 5737 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:37.560894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.14.212.126:38291)
1026 15:14:37.560942 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.560944 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.561575 (+ 631us) server_negotiation.cc:408] Connection header received
1026 15:14:37.561699 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.561701 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.561712 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.561748 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.562218 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44948: BlockingRecv error: recv got EOF from 127.0.0.1:44948 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20251026 15:14:37.563098 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:37.561309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42010 (local address 127.14.212.125:43653)
1026 15:14:37.561349 (+ 40us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.561351 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.562562 (+ 1211us) server_negotiation.cc:408] Connection header received
1026 15:14:37.562671 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.562672 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.562683 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.562714 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.563036 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42010: BlockingRecv error: recv got EOF from 127.0.0.1:42010 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20251026 15:14:37.564090 5891 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:37.563448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.14.212.124:41873)
1026 15:14:37.563487 (+ 39us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.563489 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.563590 (+ 101us) server_negotiation.cc:408] Connection header received
1026 15:14:37.563696 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.563697 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.563707 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.563736 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.564040 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45400: BlockingRecv error: recv got EOF from 127.0.0.1:45400 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20251026 15:14:37.566370 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:37.565623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.14.212.125:43653)
1026 15:14:37.565669 (+ 46us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.565671 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.565865 (+ 194us) server_negotiation.cc:408] Connection header received
1026 15:14:37.565983 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.565984 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.566001 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.566031 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.566323 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42022: BlockingRecv error: recv got EOF from 127.0.0.1:42022 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20251026 15:14:37.569478 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:37.568738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.14.212.125:43653)
1026 15:14:37.568773 (+ 35us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.568775 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.569007 (+ 232us) server_negotiation.cc:408] Connection header received
1026 15:14:37.569103 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.569104 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.569113 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.569142 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.569432 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42026: BlockingRecv error: recv got EOF from 127.0.0.1:42026 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20251026 15:14:37.573493 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:37.572719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.14.212.125:43653)
1026 15:14:37.572768 (+ 49us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.572770 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.572997 (+ 227us) server_negotiation.cc:408] Connection header received
1026 15:14:37.573094 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.573095 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.573105 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.573135 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.573444 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42036: BlockingRecv error: recv got EOF from 127.0.0.1:42036 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20251026 15:14:37.579782 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:37.578848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.14.212.125:43653)
1026 15:14:37.578890 (+ 42us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.578893 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.579153 (+ 260us) server_negotiation.cc:408] Connection header received
1026 15:14:37.579260 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.579261 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.579272 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.579303 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.579723 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42040: BlockingRecv error: recv got EOF from 127.0.0.1:42040 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20251026 15:14:37.590193 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:37.589244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.14.212.125:43653)
1026 15:14:37.589288 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.589290 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.589556 (+ 266us) server_negotiation.cc:408] Connection header received
1026 15:14:37.589680 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.589681 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.589692 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.589722 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.590121 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42048: BlockingRecv error: recv got EOF from 127.0.0.1:42048 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20251026 15:14:37.608672 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:37.607673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.14.212.125:43653)
1026 15:14:37.607721 (+ 48us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.607724 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.608033 (+ 309us) server_negotiation.cc:408] Connection header received
1026 15:14:37.608151 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.608153 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.608165 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.608196 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.608601 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42050: BlockingRecv error: recv got EOF from 127.0.0.1:42050 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20251026 15:14:37.643175 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:37.642184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.14.212.125:43653)
1026 15:14:37.642228 (+ 44us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.642230 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.642542 (+ 312us) server_negotiation.cc:408] Connection header received
1026 15:14:37.642669 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.642670 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.642682 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.642712 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.643115 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42052: BlockingRecv error: recv got EOF from 127.0.0.1:42052 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20251026 15:14:37.709389 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:37.708714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.14.212.125:43653)
1026 15:14:37.708768 (+ 54us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.708770 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.708899 (+ 129us) server_negotiation.cc:408] Connection header received
1026 15:14:37.708934 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.708936 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.708947 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.709032 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.709319 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42058: BlockingRecv error: recv got EOF from 127.0.0.1:42058 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20251026 15:14:37.839358 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:37.838644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.14.212.125:43653)
1026 15:14:37.838755 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:37.838757 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:37.838770 (+ 13us) server_negotiation.cc:408] Connection header received
1026 15:14:37.838841 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:37.838842 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:37.838852 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:37.838919 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:37.839296 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42070: BlockingRecv error: recv got EOF from 127.0.0.1:42070 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20251026 15:14:38.097666 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
1026 15:14:38.096891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.14.212.125:43653)
1026 15:14:38.096997 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:38.096999 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:38.097011 (+ 12us) server_negotiation.cc:408] Connection header received
1026 15:14:38.097132 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:38.097133 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:38.097143 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:38.097199 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1026 15:14:38.097604 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42080: BlockingRecv error: recv got EOF from 127.0.0.1:42080 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20251026 15:14:38.468865 5926 catalog_manager.cc:1269] Loaded cluster ID: e22152a97d3640889552496293cea12c
I20251026 15:14:38.468940 5926 catalog_manager.cc:1562] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905: loading cluster ID for follower catalog manager: success
I20251026 15:14:38.469851 5926 catalog_manager.cc:1584] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905: acquiring CA information for follower catalog manager: success
I20251026 15:14:38.470176 5926 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b93b0aaa97de470490732cd45976a905: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 15:14:38.515826 5949 catalog_manager.cc:1269] Loaded cluster ID: e22152a97d3640889552496293cea12c
I20251026 15:14:38.515905 5949 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3: loading cluster ID for follower catalog manager: success
I20251026 15:14:38.516745 5949 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3: acquiring CA information for follower catalog manager: success
I20251026 15:14:38.517028 5949 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6666bc18d4794cf698153614633835c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 15:14:38.613408 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:38.610939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.14.212.125:43653)
1026 15:14:38.611076 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:38.611078 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:38.611093 (+ 15us) server_negotiation.cc:408] Connection header received
1026 15:14:38.611192 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:38.611193 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:38.611234 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:38.611305 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:14:38.611667 (+ 362us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:38.612201 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:38.612812 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:38.612943 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:38.612984 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:14:38.613125 (+ 141us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:14:38.613215 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:38.613274 (+ 59us) server_negotiation.cc:300] Negotiation successful
1026 15:14:38.613296 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20251026 15:14:38.617540 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:38.615520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.14.212.124:41873)
1026 15:14:38.615623 (+ 103us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:38.615625 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:38.615632 (+ 7us) server_negotiation.cc:408] Connection header received
1026 15:14:38.615700 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:38.615700 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:38.615725 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:38.615787 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:14:38.616028 (+ 241us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:38.616510 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:38.617074 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:38.617201 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:38.617237 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:14:38.617332 (+ 95us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:14:38.617395 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:38.617447 (+ 52us) server_negotiation.cc:300] Negotiation successful
1026 15:14:38.617469 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
I20251026 15:14:38.621843 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:38.619479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.14.212.124:41873)
1026 15:14:38.619520 (+ 41us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:38.619522 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:38.619533 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:14:38.619609 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:38.619610 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:38.619621 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:38.619677 (+ 56us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:14:38.619948 (+ 271us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:38.620605 (+ 657us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:38.621391 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:38.621572 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:38.621593 (+ 21us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:14:38.621613 (+ 20us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:14:38.621661 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:38.621747 (+ 86us) server_negotiation.cc:300] Negotiation successful
1026 15:14:38.621778 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20251026 15:14:38.622153 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:38.619470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42094 (local address 127.14.212.125:43653)
1026 15:14:38.619528 (+ 58us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:38.619531 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:38.619896 (+ 365us) server_negotiation.cc:408] Connection header received
1026 15:14:38.619970 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:38.619971 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:38.619985 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:38.620036 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:14:38.620346 (+ 310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:38.620943 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:38.621650 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:38.621854 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:38.621878 (+ 24us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:14:38.621909 (+ 31us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:14:38.621958 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:38.622061 (+ 103us) server_negotiation.cc:300] Negotiation successful
1026 15:14:38.622088 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251026 15:14:38.622215 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1026 15:14:38.619539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.14.212.126:38291)
1026 15:14:38.619709 (+ 170us) server_negotiation.cc:207] Beginning negotiation
1026 15:14:38.619711 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1026 15:14:38.619722 (+ 11us) server_negotiation.cc:408] Connection header received
1026 15:14:38.619817 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1026 15:14:38.619818 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
1026 15:14:38.619853 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1026 15:14:38.619921 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
1026 15:14:38.620292 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:38.620935 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1026 15:14:38.621529 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1026 15:14:38.621740 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1026 15:14:38.621801 (+ 61us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1026 15:14:38.621921 (+ 120us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1026 15:14:38.621994 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1026 15:14:38.622088 (+ 94us) server_negotiation.cc:300] Negotiation successful
1026 15:14:38.622119 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20251026 15:14:38.624013 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 5666
I20251026 15:14:38.629513 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 5743
I20251026 15:14:38.634619 15185 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXR9jZt/build/release/bin/kudu with pid 5820
2025-10-26T15:14:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3800 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (7692 ms total)
[----------] Global test environment tear-down
[==========] 57 tests from 4 test suites ran. (426340 ms total)
[ PASSED ] 56 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
1 FAILED TEST